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Executive Summary 


The North Lawndale Cluster Initiative (NLCI) sought to improve academic achievement and educational 
attainment in a cluster of neighborhood schools in Chicago from 2013-14 through 2018-19, providing 
funding for four elementary schools and their neighborhood high school to provide vertically-integrated 
services from pre-k through high school. This evaluation describes the changes in student outcomes that 
occurred in the years of the initiative. All outcomes improved, and many outcomes showed substantial 
improvements, although only pre-k attendance was significantly higher compared to matched schools 
that served similar students. The analysis assessed the impact of funding a vertical integration cluster 
strategy in one neighborhood and should not be taken as an assessment of the effectiveness of the 
schools overall, or of individual programs. The bulk of the funding was used to expand pre-k from half- 
day to full-day programs, which showed significantly more improvement over time, relative to comparison 
schools. Outcomes in the elementary, middle, and high school grades generally improved at a similar rate 
as other district schools serving similar students. Furthermore, as described in the report, the potential 
gains that NLCI schools could have realized were limited by high student mobility rates and variability in 


the size and composition of the student population from year-to-year at the high school level. 


NLCI Overview 


The NLCI began in the 2013-14 school year. The Academy for Urban School Leadership (AUSL), a 
nonprofit school management organization, received funding from a Chicago family foundation for a 
“vertical integration” cluster strategy to improve outcomes in neighborhood schools in the North 
Lawndale area. The NLCI aimed to advance a cohesive, high-quality school and student support system by 
coordinating and aligning resources, programs, and partnerships to support students from pre-k through 
high school graduation. The largest portion of the initiative funding went to support full-day pre-k classes 
in the four elementary schools. Other supports included: family and community engagement activities; 
professional development for teachers; academic and social-emotional supports for students; a middle-to- 
high-school transition program for eighth-graders; and partnerships with other nonprofits providing post- 
secondary transition supports. This study evaluates the NLCI from pre-initiative years (2011-12 and 2012- 
13) through six years of the initiative (2013-14 through 2018-19). 


Key Findings 


il; 


Pre-k attendance improved significantly with the initiative, surpassing comparison 
schools and the district average. The NLCI provided funding for elementary schools to offer 
full-day pre-k programs instead of partial-day programs. Prior research has suggested that 
partial-day pre-k presents several barriers to students’ school attendance. In the year prior to the 
initiative, pre-k attendance was not significantly different from attendance at matched 
comparison schools; after the initiative, pre-k attendance was significantly higher. Pre-k 
attendance improved districtwide in Chicago Public Schools (CPS) but improved more at NLC 
schools. As a result, students at NLCI schools entered kindergarten having spent significantly 
more time in pre-k than both previous students in NLCI elementary schools and also similar peers 
in the district. This study provides new, compelling evidence that expanding partial-day pre-k 
programs to full-day programs can help improve attendance. 


Attendance and school climate improved over time in elementary schools, 
matching or surpassing the district average, not significantly more than 
comparison schools. NLCI schools received professional development around primary grade 
instruction in literacy and math, as well as City Year mentors to support student achievement and 
social-emotional supports for some groups of students. These were intended to improve 
students’ performance in school, and to support a stronger school climate. Attendance in grades 
K-8 was lower than the district average in pre-initiative years and similar or higher than the 
district average by 2018-19, depending on grade level. NLCI middle grade students also reported 
stronger school engagement, school connectedness, and relationships with peers on the annual 
5Essentials Surveys during the initiative than in prior years. The differences in reports on school 
climate measures were close to statistical significance in the 2015-16 school year, but otherwise 
the NLCI schools were not significantly different from comparison schools in terms of attendance 
or school climate. Thus, the improvements in attendance and climate in the elementary grades 
cannot necessarily be attributed to the initiative; the improvements likely were also influenced by 
broader district initiatives to support schools similar to the NLCI schools. 


Test Scores in grades 2-8 improved along with the districtwide improvements. 
Districtwide, test scores improved considerably over this period, such that scores at the end of the 
period were higher by one year's worth of learning in 2018-19 than in 2012-13. Test scores in 
NLCI schools improved during the years of the initiative at rates that were similar to 
improvements observed at the district as a whole. Thus, test scores improved considerably over 
this period, but the improvements in test scores cannot be attributed to the initiative. 
Additionally, test scores at NLCI schools continued to be lower than district averages. 


Students fared better in the transition to high school—reducing the gap with the 
district average but not closing it—and at similar rates as comparison schools. The 
initiative included supports for navigating the high school choice process, which should have led 


students to enroll in high schools where they were more likely to succeed. Districtwide, there were 
improvements in Freshman Ontrack rates, which often lead to higher performance when students 
transition from eighth to ninth grade. Eighth-grade graduates from NLCI schools did fare better in 
the transition to high school over time, with improving ninth-grade GPAs in their first year of high 
school. At the NLCI high school, first-time ninth-graders also were more likely to be on-track in 
their ninth-grade year. Differences in Freshman Ontrack rates relative to the district were smaller 
at the end of the initiative than the beginning but remained substantially lower than district 
averages. The NLCI Freshman Ontrack rates were not consistently higher than those at other 
schools serving similar populations of students, suggesting that the improvements largely 
resulted from other district efforts, rather than the initiative itself. 


High school outcomes improved but were lower than outcomes of similar students 
at other schools for many years. At the NLCI high school, ninth-grade attendance and 
grades improved in the two most recent cohort years. However, until the final year of the 
evaluation (2017-18), attendance and grades were lower compared to students with similar 
backgrounds and eighth-grade achievement at other high schools. College enrollment rates 
among cohorts of graduates had declined considerably in the years just prior to the initiative and 
continued to be lower than other CPS high schools, in the first two years of the initiative. They 
improved in the 2015-16 school year and then declined again. In most of the NLCI years, students 
at the NLCI high school had lower ninth-grade attendance, ninth-grade GPAs, college enrollment 
rates, and college persistence rates than students with similar backgrounds at other CPS schools. 


Improvements occurred despite variable student enrollment and should be 
interpreted within the context of a district showing substantial improvements. 
Overall, there were improvements in many of the student outcomes in the NLCI schools over time, 
although only pre-k attendance improved more at NLC! schools than at comparison schools. 
Districtwide, there were large improvements in many of the outcomes analyzed for this study, 
thus, the NLCI schools would have had to show exceptional improvements to demonstrate gains 
that were significantly higher than the district. 


We also do not know what would have happened in the absence of the initiative. It is possible 
that the schools would have shown a decline in achievement, or shown smaller improvements. As 
described in this report, the schools experienced very high rates of year-to-year mobility and the 
high school experienced large shifts in enrollment from year-to-year. These mobility rates were 
larger than those experienced by typical schools in the district and may have made it difficult for 
strategies and outcomes to build over time. 


As CPS district leadership thinks about supporting neighborhood schools, this 
evaluation highlights specific challenges. Neighborhood schools serve any students who 
live in their attendance zone, taking in students at all times of the year. They have less certainty 
than other schools about who they will serve and how many students will enroll. One substantial 


challenge is to develop the capacity for staff to be ready to serve a continually shifting population 
of students that may have different needs from one year to the next. Particularly for schools that 
historically have high student mobility rates, and where families have fewer economic resources 
for school engagement, it will be difficult to show strong performance ratings relative to schools 
that serve families with more resources. Low school performance ratings, in turn, signal to families 
that they should consider enrolling in schools outside of their neighborhood, which works against 
a cluster system model. The NLCI schools were able to show considerable improvements in some 
outcomes, but for outcomes that research has shown are often strongly associated with the 
economic backgrounds of students—test scores and college enrollment—the resources required 
to produce equitable outcomes are likely considerable. 


Introduction 


Chicago Public Schools (CPS) have shown considerable improvements in student achievement over the 
last 20 years, but concerns about unequal educational opportunities across different communities in the 
city continues to be a pressing issue. Just recently, in 2018, the district established the Office of Equity out 
of these concerns. Across the country, researchers have documented very high correlations between 
school performance levels and the economic backgrounds of the students served by the school. Chicago 
is no exception to this pattern, with the lowest-rated schools concentrated in the least-affluent areas of 
the city.! One strategy to address inequities in educational opportunity is to invest in schools in 
neighborhoods with the lowest income levels, providing additional funding to support evidence-based 
programs. This report provides a summative evaluation of one such effort, the North Lawndale Cluster 
Initiative (NLCI). 


Local and federal policymakers have enacted numerous strategies over the past several decades to try to 
rapidly improve the performance of schools with low achievement. There are multiple models for doing 
this work. One strategy that has been recommended by experts on school turnaround is to use a cluster- 
based strategy. They note that school clusters can provide schools with more support than a single school 
turnaround, allowing schools to have authority over making decisions while receiving support and 


guidance from the network-level managing organization. 


Under the Obama administration, the U.S. Department of Education supported a neighborhood-based 
approach to offset the effects of growing up in poverty through “cradle to career” supports in Promise 
Neighborhoods. The strategy being studied here shares some elements with this approach, in that it 
provided funding for many of the same types of supports—early childhood supports; in-school tutoring in 
grades K-12; professional development for teachers; college and career mentorship; and high school and 
college application assistance—all for a specific neighborhood. There were differences between Promise 
Neighborhoods and the NLCI, too: Promise Neighborhoods were intended to encompass family and 
community supports external to schools, and were not focused on particular schools, ? although, in 
practice, even the Promise Neighborhoods tended to focus mostly on in-school efforts.* The Promise 
Neighborhoods were funded just before and during the period of the NLCI, but there has been limited 
evidence on their effectiveness. The research that exists suggests mixed results in the first few years, and 
notes difficulties in obtaining data and tracking outcomes that might not be affected for at least 10 years, 


1 Chicago Public Schools (2019). 

2 Calkins, Williams, Belfieore, & Lash (2007). 
3 Hulsey et al. (2015). 

4 Bower & Rossi (2019). 


as students move through supports at different ages, as well as issues with staff turnover and student 
mobility.° 


The North Lawndale Cluster Initiative C(NLCI) 


The NLCI began in the 2013-14 school year. The Academy for Urban School Leadership (AUSL), a 
nonprofit school management organization that has a mission of transforming chronically low-performing 
schools, received funding for a “vertical integration” cluster strategy to turn around underperforming 
schools in the North Lawndale area. The NLCI aimed to advance a cohesive, high-quality school and 
student support system by coordinating and aligning resources, programs, and partnerships to support 
students from pre-k through high school graduation. 


The initiative provided support for programs in four elementary schools (Dvorak School of Excellence, 
Herzl Elementary School, Johnson School of Excellence, and Chalmers School of Excellence) and one high 
school (Collins Academy High School). As shown in Table 1, the elementary schools chosen for the 
initiative historically had low test scores relative to the district, even as they showed improving 
performance over time, relative to the state standards. In 2013, about one-quarter of students in the NLC 
schools met state standards, compared to about one-half of students in the district as a whole. Further 
information about the population of students at each of the schools is provided in the next chapter. 


The largest portion of the initiative funding went to support full-day pre-k classes in the four elementary 
schools. Prior to the initiative, each of the elementary schools offered half-day pre-k programs. Research 
has suggested that partial-day pre-k presents several barriers to students’ school attendance, and full-day 
pre-k offers many benefits.° The differential costs for offering full-day pre-k classes, rather than partial- 
day classes, ranged from $600,000 to over $1 million a year. In addition, the NLCI schools received $2 
million over three years for an array of supports that included family and community engagement 
activities; professional development for teachers (including math and early literacy professional 
development through an external grant to the Erikson Institute); academic and social-emotional supports 
for students (the LAMP mentoring program and the UCAN educational support program); a middle-to- 
high-school transition program for eighth-graders; and partnerships with other nonprofits providing post- 
secondary transition supports. There was also funding of $120,000 a year to support City Year 
programming in schools, to make up for a reduction in district funds to cover those costs. 


> Hulsey et al. (2015). 
6 Robin, Frede, & Barnett (2006); Lee, Burkam, Ready, Honigram, & Meisels (2006); Ehrlich et al. (2014). 


Table 1. The NLCI Sought to Improve Academic Performance in a Cluster of 
Schools with Historically Low Test Scores Relative to the District as a Whole 


Percent of students in 3rd-8th Grade meeting ISAT standards (Composite Score) 


School Year Chalmers Johnson Herzl Dvorak District 
2000-01 9.7% 10.8% 16.4% 17.1% 23.4% 
2001-02 8.3% 12.1% 18.1% 20.7% 25.5% 
2002-03 10.5% 13.3% 15.9% 26.3% 26.6% 
2003-04 14.7% 15.7% 17.6% 27.1% 29.4% 
2004-05 10.9% 11.3% 19.4% 26.2% 29.8% 
2005-06 18.2% 12.7% 26.0% 29.1% 374% 
2006-07 16.4% 21.6% 28.9% 29.3% 39.6% 
2007-08 17.1% 14.2% 32.2% 33.3% 39.6% 
2008-09 13.2% 17.9% 24.6% 32.1% 42.0% 
2009-10 19.7% 24.1% 17.4% 27.6% 44.8% 
2010-11 24.0% 30.7% 18.4% 32.7% 48.7% 
2011-12 25.3% 38.7% 16.2% 28.2% 50.7% 
2012-13 28.2% 21.9% 25.3% 24.0% 52.5% 


Note: Data represented here were sourced from Chicago Public Schools Assessment Reports: 
https://cps.edu/SchoolData/Pages/SchoolData.aspx. 


By providing supports at all grade levels, the initiative aimed to boost the outcomes of all students in the 
NLCI schools. Often, successful programs fail to show long-term success, as initial gains fade out over 
time. By putting resources into programs for all grades, the vertical strategy provided the potential to 
reduce any fade-out in earlier gains. One hope for the NLCI was that it would provide a strong experience 
for students from pre-k through high school, with continual improvements observed in student outcomes 
as students moved through grades, having had more years of supports at earlier grades. 


In this report, we examine the degree to which outcomes, from pre-k attendance to college persistence, 
changed over time at the NLC! schools. The supports provided at each grade level are described below 
and summarized in Table 2. The Appendix provides information about all of the outcome measures used 
in this study, each of which is also described briefly below. 


The NLCI schools historically had lower academic performance levels than the district as a whole, as 
described in Table 1. This was the main reason the district contracted with AUSL to manage the schools, 


and the initiative was started to provide extra supports—the goal was to boost achievement.’ Therefore, 
besides showing how outcomes changed in the schools, we also show how outcomes improved relative to 
the district average. This shows whether the schools were catching up to the district average, which is 
relevant for district goals around equity. At the same time, comparing the outcomes of NLCI students to 
the district average does not tell us how well the NLCI schools and the supports of the initiative were 
serving students, since students’ outcomes are influenced by factors outside of school. Therefore, we also 
conduct analyses that compare outcomes among students with similar background characteristics, and at 
schools that were similar to the NLCI schools prior to the initiative. For both the elementary schools and 
the high school, we answer three questions: 


1. Were there improvements in student outcomes in NLCI schools in the years of the initiative? 
2. Ifso, did student outcomes improve more in NLC! schools than in the district as a whole? 


3. Were outcomes at NLCI schools significantly higher than at schools serving similar students? 


Table 2. The Initiative Provided Supports from Pre-K through College 
Preparation 


NLCI supports and corresponding outcomes 


Elemen nd Middle High School - 
Grade Level: Pre-K ermentaryard Bild 8th - 9th Grade 9 
Grades College 
: . ity Year 
e Teacher professional e City Year ow = 
; mentoring 
development mentoring : : 
Full-day . : : Social-emotional 
Supports: City Year mentoring e¢ Highschool 
Pre-k ; ‘ ne supports 
e Social-emotional transition 
supports e §=Advisin eoteae 
PP g counseling 
e Attendance 
: 9th-grade e College 
e  =Pre-k e Reading and math 
Outcomes: course enrollment and 
attendance scores 


: erformance ersistence rates 
e School climate e P 


7 https://www.chicagotribune.com/news/ct-xpm-2014-05-10-ct-privatization-schools-north-lawndale-201405 10-story.html 
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Elementary School Outcomes 


We begin by showing changes over time on an array of outcomes at the four elementary schools. The 
four elementary schools are studied as a group. We do not examine variation in outcomes between the 
four schools, as the goal of this evaluation was to determine the overall impact of the initiative. Each of 
the outcome measures is described below, with further information on the outcome measures and 
analytic methods provided in the Appendix. 


Pre-K Attendance 


The initiative provided considerable resources to transition from partial-day to full-day pre-k classes in the 
four NLCI elementary schools. Prior research suggests that supporting full-day pre-k programs has good 
potential to benefit student outcomes, especially among students starting with the lowest achievement 
levels. Full-day pre-k is associated with significantly higher test scores in math and literacy, relative to half- 
day pre-k.® Studies of full-day vs. half-day kindergarten also show that students have higher growth rates 
on a variety of academic skills.? Furthermore, interviews with pre-k parents suggest that attendance rates 
at pre-k are influenced by whether the program is a full-day program or a partial day program. It can be 
more difficult for families to get students to and from school in half-day programs, as parents need to 
leave work in the middle of the day. It can also seem like it is less worthwhile to go to school if there are 
competing priorities on that day, such as transportation difficulty, a sick sibling, or other commitments. '° 
Pre-k attendance can seem unimportant, but it is predictive of students’ learning gains in pre-k, their 
kindergarten readiness, their likelihood of reading at grade level when they reach the second grade, and 
their long-term educational attainment." Investments in efforts to improve pre-k attendance may also 
improve equity in outcomes. Students who started pre-k furthest behind in terms of literacy skills are the 
most likely to be chronically absent, and absenteeism matters more for their academic growth than 
among students who started less far behind. "2 


Therefore, one key outcome for this study is pre-k attendance. Some students begin pre-k at the age of 
three, but many more attend pre-k at the age of four, so we examine attendance rates among pre-k 
students at four. We also look at the degree to which students entering kindergarten at the NLCI schools 
attended pre-k at any CPS school, and their attendance rates when they were in pre-k, as indicators of 
preparation for kindergarten. 


Outcomes studied: 
e Pre-k attendance rates among four-year-old pre-k students 


e Any prior pre-k attendance among entering kindergarten cohorts 


8 Robin et al. (2006). 

2 Lee et al. (2006). 

1 Ehrlich et al. (2014). 

" Berlinski, Galiani & Manacorda (2008); Ehrlich et al. (2014); Taylor, Gibbs, & Slate (2000). 
12 Ehrlich et al. (2014). 


Attendance and Test Scores in Pre-K-Grade 2 


Investments in pre-k might have further benefits as students move into kindergarten and the primary 
grades. Students who have higher attendance rates in pre-k tend to have higher attendance rates in 
kindergarten through second grade. Students with higher rates of pre-k attendance also begin 
kindergarten with stronger academic and social skills and have higher test scores when they get to second 
grade.'? Attendance in elementary schools has also been shown to strongly predict learning and success 
when students get to high school."4 The first year in which all NLCI schools had full-day pre-k classes was 
2014-15. If they made expected progress in school, students in pre-k during the initiative years would 
have been in second grade in the 2017-18 school year. Therefore, we examine attendance in 
kindergarten, first grade, and second grade, as well as NWEA-MAP scores in second grade, since we might 
expect greater change in those grades as a result of full-day pre-k participation in earlier years. 


Outcomes studied: 
e Attendance in kindergarten, first grade, and second grade 


e Second-grade test scores 


Attendance, Test Scores, and Perceptions of School in Grades 3-8 


In addition to expanding pre-k, the initiative included professional development for teachers, and 
coaching for students, to support achievement and student connection to school in the elementary and 
middle grade years. This included professional development for primary grade teachers around math and 
literacy through the Erikson Institute. Additionally, the NLCI schools had been partnering with City Year, 
which provided teams of AmeriCorps members to work with classroom teachers as student success 
coaches, supporting students’ social, emotional, and academic development through tutoring, school- 
wide event planning, and afterschool programs. City Year has commissioned studies in Chicago and other 
cities showing that City Year partnerships have improved student attendance, test scores, and grades." 
When the initiative was beginning, the schools were at risk for losing funding for City Year, and so the 
initiative provided funds to allow them to continue to partner. Research also highlights the need for 
supporting students’ social-emotional needs, and coaching and mentoring students when they need help. 
The City Year mentors provide support to students and teachers that might help improve school 
connectedness and academic engagement." In addition, the initiative included partnerships with several 
nonprofit organizations to provide mentorship to students in specific groups. These included the LAMP 
Amachi program, which provides caring adult mentors to children whose parents are incarcerated, and the 


13 Ehrlich et al. (2014). 

14 Allensworth, Gwynne, Moore, & de la Torre (2014a). 

15 Copeland & Raynor (2018); Chapin Hall (2017). 

18 Information about City Year approach, models, and training can be found at: https://www.cityyear.org/impact/unique- 
approach/ 


10 


UCAN educational support program, which helps to engage youth in foster care to promote education 
and build positive social supports. 


The professional development, City Year mentors, and social-emotional supports in schools were intended 
to support improvements in attendance and test scores, as well as students’ perceptions of the school 
climate. Students in grades 6-8 take districtwide 5£ssentials Surveys, which provide information on their 
experiences in school, as described further in the Appendix. 


Outcomes studied: 
e Attendance in grades 3-8 
e Scores in reading and math on the NWEA in grades 3-8 


e Students’ reports of school climate in grades 6-8,* including school connectedness, academic 
engagement, peer support, and peer relationships 


*surveys not administered to younger students 


Eighth-Grade Graduates’ Success in High School 


Students in CPS have many different options for high school, and about three-fourths of high school 
students attend a high school other than their attendance-area neighborhood high school.'” There are 
substantial differences in ninth-grade performance among students who have the same performance in 
the middle grades, based on which high school they attend.'® This suggests that students’ decisions about 
where to attend high school matter for their academic success in high school. However, it is not always 
clear what the best school choice is for a student. Students at selective schools tend to have strong 
academic outcomes, but few students have access to selective schools, and research has shown that 
students’ outcomes are sometimes better if they attend strong neighborhood schools or other academic 
programs. '? Students have differential access to high schools based on their academic records and other 
factors, such as services they may need or access to transportation. They also have different preferences 
for the types of academic and extracurricular programs offered at different high schools. Therefore, what 
makes a school a good fit for one student may not be the same for another student. Navigating all of the 
information about schools and figuring out where a student is most likely to be successful can be 
challenging.”° 


Prior to the 2017-18 school year, the process for applying to high schools was complicated, with different 
application processes and deadlines for different schools. It could also be difficult to find information 
about different high schools, let alone make decisions about which high schools to attend. The NLCI 


7 Barrow & Sartain (2019). 

18 Allensworth et al. (2014a); Gottfried (2011). 

19 Allensworth, Moore, Sartain, & de la Torre (2016); Barrow, Sartain, & de la Torre (2018); Coca et al. (2012). 

0 See forthcoming Consortium brief, Student Experience with the High School Choice Process in Chicago Public Schools 
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supported programs were intended to help students navigate the high school choice and application 
process and be ready for the transition to high school. 


If students were more likely to enroll in schools that were a better fit for them—based on their own 
backgrounds, interests and needs—we would expect to see higher ninth-grade outcomes. Therefore, we 
examine the ninth-grade GPAs of the eighth-graders who graduated from the four elementary schools, as 
an indicator of whether students’ high school outcomes improved with better support through the 
decision-making process. We examine students’ success in high school, rather than their rates of enrolling 
at particular types of schools, since there is no one clear type of high school that would be the best match 
for all students. 


Outcomes studied: 


e Ninth-grade GPAs among eighth-grade graduates 


Analytic Methods: Elementary School Outcomes 


For each outcome, we show the average across all four elementary schools, along with the district 
average. The NLCI and district averages are based on the population of students in NLCI schools and in 
the district, not on samples. Thus, all of the changes shown in the figures represent the actual changes 
that occurred, without sampling error. We also show outcomes in the year before the initiative, 2012-13, 
as a point of comparison. Data on some of the elementary school outcomes are only available beginning 
in the 2012-13 school year, so we can only show one year of pre-initiative data at the elementary school 


level.2' 


To make more rigorous assessments of whether outcomes improved more than would be expected at 
NLCI elementary schools, we used a propensity-score strategy, comparing schools that had the same 
characteristics as the NLCI schools prior to the initiative. The propensity scores were based on school 
characteristics two years prior to the start of the initiative (2011-12), described in detail in the Appendix, 
with schools that were most similar to the NLCI schools more heavily weighted in the statistical analyses 
than schools that were less similar. We show the unadjusted trends and the district average for 
comparison in the figures, while the estimates based on the propensity weights are provided in the tables 
following the figures, along with tests of statistical significance. 


21 The elementary grade assessment that covers the broadest range of years of this study is the NWEA. CPS first started 
administering the NWEA in the 2012-13 school year. Pre-k attendance is available in the 2011-12 school year, but to 
examine the pre-k attendance of kindergarten students, we need data from the prior year, which also restricts our analysis of 
that outcome to begin in 2012-13. 
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High School Outcomes 


We examined a number of high school outcomes that might reasonably be expected to improve, given 
the ninth-grade transition supports, social-emotional supports, and college supports that were included in 
the initiative. The analyses were conducted with two types of cohorts: one at the beginning of high school 
and another at the end of high school. 


Ninth-Grade Course Performance 


First, we show the outcomes of cohorts of first-time ninth-graders. The ninth-grade transition supports 
and social-emotional supports were intended to help students perform better in their ninth-grade year. 
Prior research has shown that students’ ninth-grade attendance, GPA, and Freshman Ontrack status are 
highly predictive of whether they eventually graduate, go to college, and persist in college.?2 


Outcomes studied: 
e Ninth-grade GPAs 
e Freshman Ontrack rates23 


e Ninth-grade attendance 


College Outcomes among Graduates 


Even though most students in CPS intend to get at least a four-year college degree, there are many 
different potholes on the road to college, beyond academic preparation, that cause them never to get 
there—from navigating information about colleges, to the application process, financing college, and then 
adapting to the new challenges that arise with the college transition.“4 As with the high school choice 
process, there is also evidence that where students attend college makes a difference for whether they are 
likely to succeed once there. Students with similar backgrounds and high school achievement are more 
likely to graduate if they enroll in colleges with high institutional graduation rates, which are often more 
selective and highly-resourced colleges.” For this reason, college advisors will often encourage students 
to attend the most selective colleges that “match” their qualifications, which tend to have higher 
graduation rates. Students also consider many other factors about colleges, and look for places they 
consider a good fit for their interests, needs, and background. If effective, supports around the college 
search and enrollment process were intended to lead more students to be able to enroll in college and to 
enroll in colleges where they were likely to persist. Therefore, we examine college outcomes among 
cohorts of graduates from Collins Academy High School. We examine students’ success enrolling and 


22 Allensworth & Easton (2007); Easton, Johnson & Sartain (2017). 

23 Freshman Ontrack status is an indicator of whether students made sufficient progress in the ninth grade to be likely to 
graduate high school in four years. A student is considered on-track if they failed no more than a semester of a core course 
(i.e., English, math, science, or social studies) and have sufficient credits to move into tenth grade. 

*4 Roderick, Coca, & Nagaoka (2011). 

25 Allensworth & Clark (2019); Roderick, Holsapple, Clark, & Kelley-Kemple (2018); Roderick et al. (2011). 
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persisting in college, rather than their rates of enrolling at particular types of colleges, since the types of 
colleges that are a good fit and match differ by students’ qualifications, interests and needs. 


Outcomes studied: 
e Enrollment at two-year or four-year colleges among Collins graduates 
e Enrollment at four-year colleges among Collins graduates 


e College persistence among Collins graduates who enrolled in a four-year college 


Analytic Methods: High School Outcomes 


Collins began the initiative in 2013-14. For each outcome that we studied, we show the performance of 
Collins students, starting with the 2011-12 school year, and in each subsequent year. This allows us to see 
student outcomes in the two years prior to the initiative; unlike the elementary school outcomes, data on 
the high school outcomes are available for many years prior to the initiative. 


As with the analysis of elementary grade outcomes, we show the average outcomes in Collins, as well as 
the district as a whole. We also compare the outcomes of students at Collins to the outcomes of students 
with similar incoming characteristics at other CPS high schools. The Appendix contains further details on 
the analytic methods. In many years, Collins did not serve a large number of students. Therefore, the 
number of students at Collins included for the outcome analyses is very small for some of the years and 
some of the outcomes. For example, in some years, the analysis of college persistence rates is based on as 
few as 12 students. This limits our ability to attribute changes to the initiative vs. random variation or 
other changes occurring in the district. However, we can see whether the school effects for Collins 
become more positive over time, comparing years before and after the initiative. The school and district 
trends are based on the population of students at Collins, and in the district as a whole. 
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Context of the NLCI 


During the years of the NLCI, there were many districtwide initiatives to improve achievement at all 
schools. CPS implemented a new teacher evaluation policy, based around observations and feedback of 
teachers’ practices in the classroom.*° They introduced new discipline policies that greatly reduced the use 
of suspensions.’ They started investing more in pre-k programs throughout the city, particularly in 
neighborhoods with a large proportion of pre-k-aged children living in poverty, who were also 
experiencing low rates of pre-k enrollment.”8 They introduced a new accountability system that produced 
school quality ratings based on a wider variety of metrics than in the past and provided an overall 
summative rating to each school.”? The district made investments in school leaders, and in professional 
development and resources around Common Core and Next Generation Science Standards.*° The district 
also went through budget difficulties and leadership transitions. During this time period, students in CPS 
schools showed notable improvements in achievement and educational attainment, drawing national 
attention to the progress made in the district—with achievement gains that put the district at the 95th 
percentile and improvements in high school graduation rates that surpassed national gains.*" 


All of these changes can make it difficult to evaluate the effects of the NLCI; its schools would have been 
affected by changes that were occurring more broadly in the district. Because achievement improved 
districtwide, when we compare achievement at the NLCI schools to other schools in the district, we are 
comparing them to an ever-improving baseline. Reaching district goals around equity means schools that 
started out with lower achievement would need to show gains that were even larger than the district 
average to reach overall district outcome levels. Before considering whether schools in the initiative 
showed such sizable improvements, it is helpful to understand the broader context of the NLCI schools 
within the district. 


North Lawndale Community 


The community of North Lawndale experienced economic disinvestment from 1960-2010, with 
implications for school achievement today. North Lawndale is located on the West Side of Chicago and 
has been a predominantly Black community since the 1950s. Pre-1950, it was a largely Czech, then a 
Jewish, community, and considered an economically vibrant neighborhood in the city. The population of 
North Lawndale declined significantly between 1960 and 2010, dropping from 124,937 in 1960 to 35,912 
in 2010. Redlining by banks, discrimination by the Federal Housing Authority, and predatory real estate 
and mortgage practices led to increasing segregation, declining rates of home ownership, and economic 


°6 See http://consortium.uchicago.edu/teach-eval for more details. 
27 Stevens et al. (2015). 
8 Ehrlich et al. (2020). 
29 See CPS School Quality Rating Policy at https://cps.edu/Performance/Pages/Performancepolicy.aspx 
30 See CPS resources around the Common Core Standards at: https://cps.edu/commoncore/Pages/Commoncore.aspx 
31 Reardon & Hinze-Pifer (2017); Nagaoka, Seeskin, & Coca (2017). 
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exploitation of residents.** Changes in industry resulted in high rates of unemployment and building 
vacancies.*? For example, International Harvester closed in 1969, Sears began closing in 1974, and most 
other industries closed by the 1980s. After several decades of economic disinvestment in North Lawndale, 
several efforts have been made in the last few decades to revitalize the area. The Steans Family 
Foundation has provided programs and grants for the areas beginning in the 1980s, and more recent 
efforts such as the North Lawndale Community Coordinating Council have tried to develop the area. Since 


2010, the population has stabilized, staying at around 36,000.*4 


Table 3 shows the economic characteristics of the North Lawndale Community area census tracts, relative 
to other census tracts in Chicago. North Lawndale has higher residential stability than other 
neighborhoods, among both owners and renters. However, residents have low rates of house ownership, 
with only 24.6 percent of residents owning their own house, which puts them at the 17th percentile, 
relative to census tracts across the city. Many families in North Lawndale struggle economically, as shown 
through the median family income and the percentage of families below the poverty line. The median 
family income is $25,153, which is at the 8th percentile, relative to other census tracts in the city, and the 
percentage of families below the poverty line is 44.7 percent, at the 93rd percentile. Thus, income levels in 
North Lawndale are low relative to the rest of the city, while the poverty rate is high. 


Table 3. Families in North Lawndale have High Residential Stability, but Low 
Rates of House Ownership and Income 


North Lawndale Community Area Census Tract Value Percentile 
Average number years tenancy per unit: owners 23.6 77 
Average number years tenancy per unit: renters 10.4 7/ 
Percent lived in same house in past year 85.4% 49 
Percent who own their own house 24.6% 17 
Median family income $25,153 8 
Percent below poverty line - families 44.7% 93 


Note: Values are the average of the North Lawndale Community Area Census Tracts, weighted by population. Data 
come from the 2015 U.S. Census American Communities Survey. The percentiles are relative to all census tracts in the 
city. 


The economic situation of families in North Lawndale has implications for the reported performance levels 
of schools that serve the community, and the efficacy of an initiative such as the NLCI. Numerous studies 
have shown that school performance metrics like test scores, attendance rates, and high school 


32 Coates (2014). 
33 http://www.steansfamilyfoundation.org/lawndale_history.shtml 
34 https://www.cmap. illinois.gov/documents/10180/126764/North+Lawndale.pdf 
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graduation rates are strongly related to the economic backgrounds of students served by a school.*° For 
example, an analysis based on 2019 data for schools in Illinois showed that at most schools serving very 
few low-income students, 70-80 percent of students met testing norms, while at schools that entirely 
served low-income students, it was rare for any schools to have 70 percent of students meeting norms 
(see Figure 1). Thus, the lowest-performing schools that served few low-income students had achievement 
that was still higher than most of the highest-performing schools serving many low-income students. 
Furthermore, that analysis did not take into account that the highest-performing schools for students of 
all backgrounds are those that admit students selectively, based on prior achievement. 


These differences are important to recognize because they indicate that our current educational systems 
produce inequitable educational outcomes based on family income. Even schools that are doing an 
exceptional job serving their students still may be perceived to be of low quality, based on publicly- 
reported statistics about average student achievement. As a society, we do not yet know what level and 
types of resources are needed to compensate for the considerable structural inequities that produce these 
patterns in schools.*° For initiatives seeking to undo these patterns, the investments that are needed may 
be considerable, and they need to support practices that have the most leverage for producing change. 


Figure 1. School Test Score Averages are Strongly Related to the Income of 
Families Served by the School 


Percent of students meeting or exceeding ISAT standards by percentage low income among schools 
in Illinois 


2019 Elementary Schools 2019 Elementary Schools 


% ELA Proficiency 
% Math Proficiency 


60 70 80 90 100 


o 10 20 30 40 50 
% Low Income 


Note: School percent low income is correlated with ELA proficiency at -.77 and math proficiency at -.79, where zero is 
no relationship and -1.0 is perfectly related. 


35 Reardon (2011); Reardon (2019); Palardy (2013). 
36 National Academies of Sciences, Engineering, and Medicine (2017). 


17 


NLCI School Enrollment & Student Mobility 


NLCI schools experienced high student mobility, but served similar students over time. The four NLCI 
elementary schools served students with similar backgrounds before and during the initiative (see Table 
4). The total number of students in NLCI schools was somewhat higher during the initiative years than in 
the pre-initiative years, ranging from 1,767 students to 2,115 students. But the schools were almost evenly 
split between male and female students, and the racial composition of the schools was about 98 percent 
Black throughout the years. The percent of students receiving free or reduced-price lunch was around 98 
percent and the percent of students identified as diverse learners was around 12 percent in all years. 
Students came from census block groups (areas of about one city block) where the average poverty rate 
was over a standard deviation above the mean. This means that the typical student lived in a census block 
group with poverty rates at about the 90th percentile, relative to other census block groups in the city. 


Table 4. Characteristics of Students at NLCI Schools (Dvorak, Herzl, Johnson, 
and Chalmers) Stayed the Same both Before and During the NLCI 


Average 
School number Percent Percent siaanae Jvabanle Neighborhood 
of Free/Reduced Diverse 
Year Male Black . Poverty Level 
Students -Price Lunch Learners : 
(Standardized) 
Pre-NLCI 2011-12 1,798 49. A% 98.1% 98.2% 12.2% 1.284 
Years: 2012-13 1,894 49.3% 98.1% 98.2% 12.1% 1.288 
2013-14 2,036 49. A% 98.1% 98.2% 12.2% 1.292 
2014-15 1,917 49. A% 98.0% 98.2% 12.3% 1.292 
NLCI 2015-16 2,115 49.7% 98.1% 98.1% 12.5% 1.284 
Years: 2016-17 2,070 49.6% 98.0% 98.1% 12.5% 1.296 
2017-18 1,906 49.6% 98.0% 98.1% 12.5% 1.304 
2018-19 1,767 49.6% 98.0% 98.1% 12.5% 1.316 


Note: Neighborhood poverty level was calculated based on Census information about the census block on which 
each student lives. All neighborhoods across Chicago were standardized, such that zero represents the average 
neighborhood in Chicago and a change of +/- 1 represents a neighborhood above/below the average by 1 standard 
deviation. 


While the characteristics of students in the NLCI remained similar over time, there was constant change in 
the individual students in the school, due to high rates of student mobility. Despite higher rates of 
housing stability in North Lawndale, schools in the initiative had less stable student enrollment than was 
typical in CPS. This was true before and during the NLCI years. For example, in each year from 2011 
through 2016, NLCI elementary schools retained about 90 percent of their students, between September 
and May, and around 80 percent of their students from May to September (see Table 5). Combined, they 
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retained around 70 to 75 percent of their students from one year to the next. This means they lost 25 to 
30 percent of their students each year, while new students entered from other schools. Most of the 
students who left transferred to other CPS schools. For example, of the NLCI students in fall 2015 who 
were no longer at their schools in fall 2016, 48 percent transferred to another neighborhood CPS school, 
21 percent transferred to a CPS charter school, 5 percent transferred to a CPS magnet school, 19 percent 
moved outside of Chicago, and the rest transferred to other districts or private schools. 


Some mobility is normal and expected. Nationwide, only about 55 percent of students are enrolled in the 
same school from kindergarten to third grade.” But at the NLCI schools, transfer rates were higher than 
average. Compared to other CPS schools, they were at about the 25th percentile in terms of within-year or 
year-to-year stability in student enrollment. In other words, 75 percent of CPS schools had more stable 
enrollment rates than schools in the NLCI. The more that the student population changes from year-to- 
year, the more effort that school staff, families, and students need to put into building new relationships. *® 


Table 5. Student Enrollment Was Less Stable in NLCI Schools than the Median 
CPS School 


Stability rates of NLCI schools 


Year 2011 2012 2013 2014 2015 2016 
September-May Stability 

Value 89.4% 90.0% 88.1% 85.0% 88.6% 89.3% 

Percentile (within CPS) 24 27 24 14 26 29 
May-September Stability 

Value 81.9% 80.3% 82.8% 79.5% 86.5% 82.4% 

Percentile (within CPS) 27 22 21 11 40 24 

September-September Stability 
Value 73.1% 73.6% 74.7% 68.9% 77.0% 74.8% 
Percentile (within CPS) 25 26 29 14 34 29 


Note: Values for September-May stability and September-September stability are the average across the four NLCI 
schools, weighted by initial September enrollment. September-September stability is the proportion of students in 
September that remained enrolled the following September. Values for May-September stability are the average 
across the four NLCI schools, weighted by May enrollment. May-September stability is the proportion of students in 
May that remained enrolled in September. Lower percentiles mean lower rates of stability relative to other CPS 
schools. A percentile of 24 means that approximately 24 percent of CPS schools have lower rates of stability and 76 
percent of CPS schools have higher rates of stability. 


37 Burkam, Lee, & Dwyer (2009). 
38 Rumberger (2003). 
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Low stability rates have implications for the NLCI “vertical integration” cluster strategy to work as 
intended, with investments in pre-k having continued effects later in the elementary grades. Most 
students did not stay in the pipeline from lower grades to higher grades. As a result, only about one-third 
of the 2014-15 NLCI pre-k cohort (the first full-day pre-k cohort that existed at all four NLCI schools) was 
still enrolled at NLCI schools in second grade in 2017-18. As shown in Figure 2, 86 of the 199 NLCI pre-k 
students in 2014-15 were at other CPS schools by spring 2018, and 41 students had left the district. 
Among the 174 second-graders at NLCI schools in spring 2018, only 72 students had been at a NLC 
school for pre-k, with 69 students at the same school for both pre-k and second grade. This high student 
mobility limits the positive impact that we may expect to see from the full-day pre-k program carrying 
over to second grade, the first year that students take the NWEA-MAP assessment in reading and math. If 
only one-third of the students remained, any benefits from the pre-k supports that persisted through 
second grade would be reduced by two-thirds when examining second-grade outcomes. 


Figure 2. NLCI Schools Experienced High Rates of Mobility between Pre-K and 
Second Grade 


Pre-school-to-2nd grade mobility at NLCI schools 
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Note: The first two bars are based on all students who enrolled in an NLCI pre-k at age four at the beginning of the 
2014-15 school year. The third bar is based on all second-graders enrolled in an NLCI school in the spring of the 
2017-18 school year. 
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Collins Academy High School Student Body 


The background characteristics of students at Collins differed from those in the NLCI elementary schools. 
The student body at Collins was similar to the NLCI elementary schools in terms of racial composition and 
neighborhood poverty rates. However, a much larger percentage of high school students were identified 
as diverse learners: 20-30 percent of incoming ninth-graders were diverse learners during the initiative 
years, compared to 12-13 percent of students in the NLCI elementary schools in the same years. In some 
years, the incoming ninth-grade classes at Collins also had a larger proportion of male students than 
female students. 


The background characteristics of students at Collins also changed considerably from one ninth-grade 
class to another during the years of the initiative, and were qualitatively different from those in the years 
prior to the initiative. As shown in Table 6, the number of students in the ninth-grade class varied from 63 
students to 134 students during the initiative years, with large changes sometimes occurring from one 
year to the next. 


The percentage of students identified as diverse learners grew over time, from about 15 percent in the 
year prior to the initiative to 30 percent in the 2017-18 school year (see Table 6). Students entering the 
ninth grade at Collins during the initiative years had lower incoming math scores than in the pre-initiative 
years. Prior to the initiative, incoming students’ math scores ranged from 0.18 to 0.43 standard deviations 
below the district average—these values mean the typical student in those years had math scores around 
the 33rd to 42nd percentile in the district. In the years after the initiative, incoming students’ math scores 
ranged 0.58 to 0.72 standard deviations below the district average, which means the typical student had 
incoming scores that were at around the 25th percentile in the district. The two constant characteristics of 
the student body were the racial composition of the school, which was over 95 percent Black in each year, 
and the degree of poverty in students’ neighborhoods. Like the NLCI elementary schools, students came 
from census block groups where the poverty rate was over one standard deviation above the mean, 
indicating high levels of poverty. 


There were also considerable changes over time in the background characteristics of Collins’s graduating 
classes, both in terms of the number of students and the percent identified as diverse learners (see Table 
7). The graduating class ranged in size over the initiative years from 68 students in the first year to 45 
students in the second year. The percentage of graduates identified as diverse learners was about one- 
quarter of the graduating class in the first year of the initiative, but only about 9 percent in the 2016-17 
year. The graduating cohorts tended to be smaller than the ninth-grade cohorts and were comprised of a 
larger percentage of young women than young men. This occurred largely because young men were less 
likely to graduate from high school than young women. It is a pattern not only at Collins, but at most high 
schools in the district and across the country. While young men comprised about one-half or more of 
ninth-grade cohorts at Collins, young women made up close to two-thirds of the graduating classes in 
most years. 
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Table 6. Collins’s First-Time Ninth-Grade Cohorts Varied in Size and Had Higher 
Percentages of Students Identified as Diverse Learners, Over Time 


Average Average 8th- 
Number Percent Neighborhood Grade Math 
School _ of Percent Percent Diverse Poverty Level Score 
Year Freshmen’ Male Black Learners (Standardized) (Standardized) 
2011-12 93 55.9% 97.8% 17.2% 1.14 -0.43 
Pre- 
NLC] 2012-13 107 53.3% 98.1% 16.8% 1.10 -0.18 
Years 2013-14 88 52.3% 100.0% 14.8% 1.25 -0.43 
2014-15 134 53.7% 97.0% 21.6% 1.13 -0.72 
NLCI 2015-16 70 47.1% 100.0% 25.7% 1.16 -0.72 
Years 2016-17 65 56.9% 100.0% 21.5% 1.15 -0.67 
2017-18 63 49.2% 95.2% 30.2% 1.27 -0.58 


Note: Neighborhood poverty level was calculated based on Census information about the census block on which 
each student lived. All neighborhoods across Chicago were standardized, such that zero represents the average 
neighborhood in Chicago and a change of +/- 1 represents a neighborhood above/below the average by 1 standard 
deviation. Eighth-grade ISAT/NWEA math scores were standardized within each year such that zero represents the 
district average. 


Table 7. Graduating Classes at Collins Tended to be Much Smaller than the Ninth- 
Grade Cohorts and had a Larger Percentage of Young Women 


Average 8th- 


Number Percent Average Grade Math 
School _ of Percent Percent Diverse Neighborhood Score 
Year Graduates Female Black Learners Poverty Level (Standardized) 
on 2010-11 84 61.4% 97.6% 8.4% 1.11 -0.12 
NLCI 2011-12 79 44.3% 98.7% 19.0% 1.06 -0.34 
met | oae3 72 51.4% 100.0% 19.4% 1.20 -0.42 
2013-14 68 58.8% 98.5% 23.5% 1.10 -0.54 
Nucl 2014-15 45 66.7% 95.6% 11.1% 1.17 -0.37 
Years 2015-16 64 60.9% 96.9% 17.2% 1.05 -0.26 
2016-17 46 63.0% 100.0% 8.7% 1.19 -0.12 


Note: Neighborhood poverty level was calculated based on Census information about the census block group on which 
each student lives. All neighborhoods across Chicago were standardized, such that zero represents the average 
neighborhood in Chicago and a change of +/- 1 represents a neighborhood above/below the average by 1 standard 
deviation. Eighth-grade ISAT/NWEA math scores were standardized within each year such that zero represents the 
district average. 
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Elementary School Outcomes 


Initiative support for the four elementary schools came in phases; two of the elementary schools received 
funding for full-day pre-k classes in the 2013-14 school year, while two others received support beginning 
in 2014-15. The initiative also provided programs to support students through mentorship by City Year 
volunteers and social-emotional programs. In addition, teachers received professional development 
around math and literacy instruction in some of the schools, which was intended to further improve 
student learning in those areas. These programs were intended to influence a number of student 
outcomes, including attendance, test scores, high school choice, and school climate. 


For each outcome, we show the average for the NLCI schools as a whole, and for the district. These 
averages are based on the population of students in the NLCI schools and in the district; there is no 
margin of error. We then show the results of the statistical analyses that compare the outcomes at NLC 
schools to the outcomes at matched comparison schools. They provide an estimate of whether outcomes 
at NLCI schools were higher or lower than outcomes among students at other CPS schools that were 
similar to the students from NLCI schools, which is a fairer and more meaningful comparison than the 
comparison to the district average. They include tests of statistical significance, which indicate whether the 
differences relative to similar students at similar schools are likely to occur by chance. Because of the small 
number of schools in the initiative, random differences could occur easily, so that there is little statistical 
power to find differences to be significant. Therefore, we discuss the substantive size of the difference, as 
well as whether it is statistically significant. (See the Appendix for details on the statistical models.) 


Pre-K Attendance 


Pre-k Attendance Improved Over Time in NLCI Schools 


In 2012-13, elementary schools that would eventually join the NLCI had average pre-k attendance rates of 
87 percent, which was similar to the district average (see Figure 3). In the first year of the initiative, when 
two of the NLCI schools expanded to full-day pre-k, pre-k attendance rates at the NLCI schools increased 
to 91 percent. Pre-k attendance rates also improved districtwide that year, but not as much as in NLC 
schools. Attendance rates at NLCI schools remained at 91 percent in the second year of the initiative, 
when all four schools received funding for full-day pre-k programs. They improved to 92 percent in the 
third year of the initiative and remained at 92 percent for two more years. Pre-k attendance rates 
improved districtwide during this period, but NLCI pre-k attendance remained higher throughout the 
years of the initiative. However, pre-k attendance at NLCI schools declined to 91 percent in 2018-19, while 
it did not decline districtwide. 
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Figure 3. NLCI Pre-K Attendance Increased the First Year of the Initiative and 
Remained Higher than the District Average 


Pre-k attendance rates among four-year-olds at NLCI schools and districtwide 
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Note: Percentages are based on all students who enrolled in an NLCI pre-k school or CPS pre-k school at age four in 
the fall of each year. 


Higher attendance rates in pre-k are associated with better kindergarten readiness, and a higher 
likelihood of regular attendance when students get to kindergarten and the primary grades.*° However, 
not all students who attended kindergarten in NLCI schools attended pre-k in an NLCI school, and some 
students entering kindergarten did not attend pre-k at all in CPS. Therefore, we also looked to see 
whether students entering kindergarten in NLCI schools spent more time in pre-k in the prior year after 
the initiative than before the initiative. 


The overall percentage of entering kindergarteners in NLCI schools who had attended any pre-k in CPS 
increased over time, from 56 percent in 2012-13 to 78 percent in 2018-19 (see Figure 4). The amount of 
pre-k that those students experienced was also greater over time, as there were improvements over time 
in the pre-k attendance rates of entering classes of kindergarten students at NLCI schools (see Figure 5). 
Kindergarteners entering the NLCI elementary schools in 2012-13 had average attendance rates in pre-k 
of 85 percent. This rate gradually improved over time, reaching 92 percent in 2018-19. These 
improvements were larger than those observed in the district as a whole. Thus, students entering 


39 Ehrlich et al. (2014). 
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kindergarten at NLCI schools were increasingly likely to have attended CPS pre-k, and to have had higher 


attendance rates while in pre-k. 


Figure 4. An Increasing Proportion of Kindergarteners in NLCI Schools Had 
Attended a CPS Pre-K 


Enrollment at any pre-k school among kindergarteners in NLCI schools 
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Note: Percentages are based on all students who enrolled in kindergarten in an NLCI or CPS school in the fall of each 


year. 


Figure 5. Kindergarteners’ Pre-K Attendance Rates Gradually Improved Over 


Time 


Kindergarteners' attendance rates when in pre-k 


92% 


Pre-K Attendance Rate 


2012-13 2013-14 2014-15 2015-16 2016-17 2017-18 2018-19 


School Year 


m= NLCI Schools ==District 


Note: These figures are based on all students who enrolled in kindergarten in an NLCI or other CPS school in 
the fall of each year who also attended a CPS pre-k school in the prior year. 


92% 92% 


25 


Pre-k Attendance was Significantly Higher than Comparison Schools 


In the year prior to the initiative (2012-13), pre-k attendance rates at NLCI schools were not different from 
pre-k attendance rates at comparison schools; the coefficient representing the difference was close to zero 
(-0.9) and not significant (see Table 8). In the first year of the initiative, when two of the NLCI schools 
started offering full-day pre-k, attendance rates were higher in NLCI schools than in comparison schools 
by 3.8 percentage points, but the difference was not statistically significant. In the second year of the 
initiative, the pre-k attendance rates of NLCI schools remained about the same as in the prior year (about 
91 percent), but pre-k attendance rates in matched comparison schools declined. Thus, the difference in 
attendance rates in that year relative to matched comparison schools was larger (5.5 percentage points) 
and statistically significant. Pre-k attendance rates in NLCI schools improved the following year and were 
about 4 percentage points higher than at comparison schools through the 2017-18 school year, a 
difference that was statistically significant despite the small number of schools in the initiative.” 


The combination of more students enrolling in pre-k in NLCI schools, and higher attendance rates among 
students enrolled in pre-k, meant that students entered kindergarten having spent more time in pre-k in 
later years, relative to earlier years. From 2014-15 forward, NLCI kindergarteners began school with 
significantly higher pre-k attendance than their comparison group peers in each subsequent year. If we 
only compare students who attended pre-k, we see that NLCI kindergarteners had higher attendance 
rates in pre-k than kindergarteners at comparison schools who had attended pre-k, by 3.7-5.2 percentage 
points (see Table 8).The last two columns in Table 8 compare overall time in a CPS pre-k among students 
entering kindergarten, where students who did not attend a CPS pre-k have a value of zero. Including 
students who did not enroll at all in a CPS preschool, the differences in the average percent of time in a 
CPS pre-k school among kindergarteners was considerably higher in the initiative years, by as much as 24 
percentage points in the 2017-18 school year. Because students could have attended pre-k outside of 
CPS, this is likely an overestimate of the total effect on any pre-k attendance. The true effect is in- 
between the middle and last columns, but both show a large and significant effect of being at an NLC 
school. 


40 These were the conclusions of models run with data that did not include the 2018-19 school year. Adding data from the 
most recent school year resulted in changes in the coefficients and significance levels in past years due to changes in the 
relationships of the covariates with the outcome over time, as described in the Appendix. As it would not be appropriate to 
evaluate schools in earlier years based on the relationships observed among covariates in later years, we do not include 
2018-19 data in the models summarized here, but the coefficients are available in the Appendix. The estimates of pre-k 
attendance rates among kindergarten cohorts remain statistically significant even with the addition of 2018-19 data. 
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Table 8. Pre-K Attendance Rates Were Significantly Higher than Comparison 
Schools 


Coefficients from propensity-weight models with demographic controls and year fixed effects 


NLCI vs. NLCI vs. NLCI vs. 

SchaslVear NLCI Comparison NLCI Comparison NLCI Comparison 
Average __ Difference (in Average Difference (in Average Difference (in 

percentile points) percentile points) percentile points) 
2012-13 86.9% -0.9 85.3% 3.7 47.9% 15.3* 
2013-14 91.1% 3.8 87.6% 3.4 52.7% 13.0 
2014-15 90.5% 5.5* 91.3% 5.2** 60.4% 18.4** 
2015-16 91.8% 4.3* 90.1% 3.7* 62.7% 20.2** 
2016-17 92.0% 4.5* 92.1% 4.5*** 63.7% 18.0** 
2017-18 92.0% 4.3* 92.3% 44** 69.7% 23.7*** 


Note: Asterisks indicate the estimate of the difference relative to comparison schools is statistically different from zero 
(*<0.05; **<0.01; ***<0.001). Coefficients come from hierarchical linear models with students nested within schools, 
controlling for year fixed effects, whether the school was managed by AUSL, and student background (i.e., gender, race, 
ethnicity, neighborhood poverty, disability status), with propensity weights applied based on school characteristics in 
the 2011-12 school year. 


Attendance in Grades K-8 


Attendance Rates in Grades K-8 Caught Up to District Averages, and 
Surpassed Them in Some Grades 


From 2012-13 through 2014-15, attendance rates in grades K-8 were lower in NLCI schools than the 
district averages (see Figure 6). Attendance rates in the NLCI schools improved in all grade levels 
beginning in the second year of the initiative (2014-15) and continued to improve or hold steady through 
the 2017-18 school year. Attendance also improved districtwide, but the improvements were larger at 
NLCI schools than in the district as a whole. As a result, while NLCI schools started out with lower 
attendance than the district average, their attendance rates were similar to the district average, or slightly 
higher than the district average, from the 2015-16 school year forward. Overall, the increases in 
attendance rates at NLCI schools and comparison schools during this time period ranged from 3-5 
percentage points. This may sound small, but attendance is a metric where 90 percent is chronically 
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absent and 100 percent is perfect. A 3-percentage-point gain is an increase of more than a standard 


deviation.” 


Increases in attendance in the elementary grades were largest in kindergarten and first grade. At the 


beginning of the period, attendance rates were much lower in kindergarten and first grade than the other 


grade levels, as well as in comparison to the district averages. Over time those differences narrowed so 
attendance at the NLCI schools was not strongly defined by students’ grade level, as it had been in prior 


years. 


Figure 6. Attendance Improved Over Time at NLCI Schools 


Average attendance rates at NLCI and district schools in grades K-8 
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Note: Percentages are based on all students enrolled in an NLCI or CPS school in grades K-8 in the fall of each year. 


The NLCI averages are weighted by school size. 


41 In the 2012-13 school year, the standard deviation of elementary school attendance rates was 2.56 percent. 
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Attendance Rates in Grades K-8 Were Not Significantly Different than 
Matched Comparison Schools 


While attendance improved in NLCI schools, attendance rates in the elementary schools were not 
significantly higher than in the comparison schools either before or after the initiative (see Table 9). There 
were improvements in attendance at schools similar to NLCI schools during the same time period. 
Therefore, it is difficult to say whether the improvements occurred as a result of the initiative, or other 
changes occurring in the district. 


Table 9. Attendance Rates in Grades K-8 Were Not Significantly Higher at NLCI 
than Comparison Schools 


Coefficients from propensity-weight models with demographic controls and year fixed effects 


NLCI vs. Comparison NLCI vs. Comparison 
School Year NLCI Average NLCI Average . 

Difference Difference 
2012-13 0.912 0.012 0.924 -0.007 
2013-14 0.917 -0.014 0.924 -0.020 
2014-15 0.933 -0.001 0.937 -0.002 
2015-16 0.943 -0.005 0.947 -0.004 
2016-17 0.946 0.005 0.952 0.008 
2017-18 0.958 0.003 0.955 0.006 
2018-19 0.940 -.001 0.944 -0.001 

and Grade Grade 8-8 

2012-13 0.939 0.002 0.939 -0.014 
2013-14 0.931 -0.015 0.934 -0.021* 
2014-15 0.937 -0.009 0.945 -0.005 
2015-16 0.956 0.002 0.956 -0.003 
2016-17 0.959 0.009 0.953 -0.002 
2017-18 0.957 0.004 0.963 0.004 
2018-19 0.948 0.001 0.948 -0.004 


Note: Asterisks indicate the estimate of the difference relative to comparison schools is statistically different from zero, 
p*<0.05. Coefficients come from hierarchical linear models with students nested within schools, controlling for year 
fixed effects, whether the school was managed by AUSL, and student background (i.e, gender, race, ethnicity, 
neighborhood poverty, disability status), with propensity weights applied based on school characteristics in the 2011- 
12 school year. 
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Test Scores in Grades 2-8 


Elementary Students’ Test Scores Improved Over Time 


Designers of the initiative hoped that improvements in pre-k attendance would eventually lead to 
improvements in achievement test scores in the elementary grades, since students would begin 
kindergarten better prepared for school. Full-day pre-k was available at all four NLCI schools in 2014-15, 
so those students would be in second grade in 2017-18 (if they were not held back), which is the first 
grade at which CPS students take the NWEA-MAP assessment in reading and math. Thus, the only 
improvement we might expect to see in NWEA scores as a result of the full-day pre-k funding would be in 
second-grade scores in 2017-18 and 2018-19. Therefore, we separate out second-grade test scores to 
examine change over time at NLCI schools when examining achievement. We might also expect to see 
improvements in math and reading scores in earlier years at other grades from the investments in teacher 


professional development. 


As shown in Figure 7 and Table 10, NWEA scores in reading and math gradually improved in NLCI 
schools over the years of the initiative. In the second grade, slight improvements began in the 2013-14 
school year, and scores either improved or remained steady for each of the following years through the 
2017-18 school year. The size of the improvements was not greater in the last two years than in prior 
years, and second-grade scores even declined slightly in the 2018-19 school year. Thus, there is little 
indication that improvements in pre-k attendance subsequently benefitted second-grade test scores three 
years later. 


NWEA reading and math scores in grades 3-8 also improved over time in the NLCI schools. Math scores in 
grades 3-8 improved in the 2014-15 school year, and then remained fairly steady, while reading scores 
improved gradually over time. The improvements were fairly sizable: in grades 3-8, math scores were 
about 0.4 standard deviations higher in 2018-19 than in 2012-13, and in reading, scores were 0.32 
standard deviations higher. In general, across the country, student achievement improves by less than 
0.40 standard deviations over the course of an academic year.** Thus, these gains suggest that students at 
NLCI schools were achieving an extra year’s worth of learning by the end of the period, relative to their 


achievement in earlier years. 


42 Bloom, Hill, Black, & Lipsey (2008). 
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Elementary Students’ Test Scores Were Not Significantly Different than 
Comparison Schools in Any Year 


Table 10 also shows the degree to which average test scores in NLCI schools were different from those of 
comparison schools. The differences are generally small, and none are statistically significant. In addition 
to comparing average scores, we also examined the yearly gains students made on NWEA scores, testing 
whether the size of the gains was larger at the NLCI schools than at comparison schools (see Table 11). 
These were also similar between NLCI schools and the comparison group. In almost all years after the 
initiative started (2013-14), there were no significant differences in test score gains between NLCI schools 
and the comparison group. For this analysis, we examined each grade level separately, resulting in many 
different grade-by-year combinations. NLCI schools and the comparison group differed in their gains only 


in about 6 percent of instances; these were rare and seemed to be random. 


Figure 7. NWEA Math and Reading Scores Increased Districtwide and in NLCI 
Schools 


Average NWEA math scores Average NWEA reading scores 
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Note: Test score averages are based on all students enrolled in an NLCI or CPS school in grades 2-8 in the spring of 
each year. 
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Table 10. NWEA-MAP Math and Reading Scores Were Not Significantly Different 
in NLCI and Comparison Schools 


NWEA scores, second to eighth grade 


2nd-Grade Grades 3-8 

School NLCI Average esa cen NLCI Average Saeed 

Meat prMenvereye Difference Estimate BE Seere Difference Estimate 
2012-13 180.7 -0.6 207.1 -2.2 
2013-14 182.4 1A 207.7 -3.4 
2014-15 183.0 -1.0 212.0 -1.9 
2015-16 185.6 -0.1 212.7 -2.3 
2016-17 185.8 0.1 212.9 -2.0 
2017-18 185.7 0.4 213.2 -2.1 
2018-19 183.4 -1.9 213.6 -2.1 

Reading 

2012-13 174.6 -0.8 199.7 -0.8 
2013-14 175.9 -0.2 201.0 -1.6 
2014-15 175.4 -3.2 204.2 -0.4 
2015-16 178.0 -2.9 203.7 -2.1 
2016-17 178.4 -2.4 203.8 -2.5 
2017-18 180.0 0.7 204.2 -1.6 
2018-19 178.8 -1.1 205.9 -0.6 


Note: None of the differences with comparison schools were significantly different from zero. NLCI comparison 
coefficients come from hierarchical linear models with students nested within schools, controlling for year fixed 
effects, grade level, whether the school was managed by AUSL, and student background (i.e., gender, race, ethnicity, 
neighborhood poverty, disability status), with propensity weights applied based on school characteristics in the 2011- 
12 school year. 
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Table 11. NWEA-MAP Test Score Gains Were Not Significantly Different between 
NLCI and Comparison Schools in Most Grades and Years 


Differences in NWEA gains at NLCI schools relative to district average gains for similar students at 
similar schools by grade and school year 


Difference in NWEA Gains Relative to District Average 


ig 3rd-Grade 4th-Grade 5th-Grade 6th-Grade 7th-Grade 8th-Grade 
2013-14 -2.435 0.0786 1.746 0.64 -1.821 -4.441 
2014-15 2.009 1.901 1.603 1.851 0.151 0.478 
2015-16 -0.924 -0.0226 0.827 1.925 -0.594 1.672 
2016-17 -1.748 1.824 0.774 -3.438 -0.599 1.864 
2017-18 -2.536 1.742 1.609 -1.329 -0.131 -4,527** 
2018-19 -0.798 -0.127 -3.044 -0.379 -1.769 -0.435 


Difference in NWEA Gains Relative to District Average 


sii 3rd-Grade 4th-Grade 5th-Grade 6th-Grade 7th-Grade 8th-Grade 
2013-14 -3.094 0.901 1.012 -1.143 -0.735 -2.52 
2014-15 2.928 4.648** 0.466 -1.565 0.006 1.299 
2015-16 -0.686 1.386 -1.136 -0.836 -0.403 -2.304 
2016-17 -1.831 2.957 -1.234 -2.594 -1.98 -0.951 
2017-18 -1.671 -0.003 1.732 2.537 0.734 -0.923 
2018-19 0.451 1.285 0.506 0.949 -1.371 -0.855 


Note: The differences are on the original NWEA score point scale. Positive differences indicate higher gains for NLCI 
schools; negative difference indicate lower gains for NLCI schools. Asterisk(s) indicate the estimate is statistically 
different from zero (*<0.05; **<0.01; ***<0.001). Coefficients come from hierarchical linear models with students 
nested within schools, controlling for year fixed effects, whether the school was managed by AUSL, and student 
background (i.e., gender, race, ethnicity, neighborhood poverty, disability status), with propensity weights applied 
based on school characteristics in the 2011-12 school year. 
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Eighth-Grade Graduates’ High School Performance 


The initiative included supports to facilitate students’ transition from eighth to ninth grade, including 
guidance on how to select and enroll in a high school within Chicago's system of school choice. Thus, to 
evaluate the NLCI, we wanted to use some indication of the quality of the transition from eighth to ninth 
grade for students who graduated from NLCI elementary schools. The “vertical” design of the initiative 
initially assumed students would graduate from the elementary schools and then enroll in the NLCI high 
school. However, it became clear after talking with adults and students in the NLCI schools, that many 
didnot consider the NLCI high school (Collins) to be the best option for all students. Students, families, 
and adults in the elementary schools who were advising students considered many different factors when 
choosing a high school, including but not limited to the performance ratings of the school. For some 
students, the neighborhood proximity of the NLCI high school made it a strong consideration, while 
others looked elsewhere. For many students, different academic and extracurricular programs made other 


schools more attractive alternatives. 


Because the characteristics that determine the best “fit” of a high school differs by students, we decided to 
examine how students performed in their first year of high school to evaluate the high school transition 
supports that were part of the initiative. To do this, we used students’ ninth-grade GPAs the year after 
they completed eighth grade in an NLCI school, as a measure of the success of the transition supports. 
The reasoning was that if eighth-grade graduates were making better choices for themselves, we should 
see that they had stronger course performance in their first year of high school than previous cohorts of 
eighth-grade graduates at NLCI elementary schools, or similar students at matched comparison schools. 


NLCI Eighth-Grade Graduates Showed Improved High School GPAs over 
Time 


NLCI graduates’ ninth-grade GPAs improved over time, from 1.86 for the 2012-13 graduating cohort to 
2.20 for the 2016-17 graduating cohort (see Figure 8 and Table 12). The increase in average GPA of 0.34 
GPA points means that in later NLCI cohorts, the average student started out high school with a solid C 
average, rather than a C-/D+ average. While this may sound like a small difference, prior research has 
shown that students with a D+ ninth-grade GPA are 20 percentage points less likely to graduate high 
school than students with a solid C GPA.*? Ninth-grade GPAs also improved across the district during this 
period, by 0.21 GPA points. While ninth-grade GPAs improved more among NLC! school eighth-grade 
graduates, they remained considerably below the district average in all years. 


43 Allensworth & Easton (2007). 
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Figure 8. Ninth-Grade GPAs Gradually Increased for NLCI Eighth-Grade 
Graduating Cohorts 


Average ninth-grade GPA of eighth grade graduates from NLCI schools and district 
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Note: Years represent the eighth-grade year at NLCI elementary schools. Ninth-grade GPAs are calculated from the 
following year. The numbers are based on all eighth-grade graduates from NLCI or CPS schools who enrolled in ninth 
grade in the district the following year. 


Ninth-Grade GPAs among NLCI Eighth-Grade Graduates Were Not 
Significantly Different than the Comparison Group in Any Year 


Average ninth-grade GPAs among graduates of NLCI schools were similar to those at comparison schools 
in every year (see Table 12). There were some years in which ninth-grade GPAs at Collins differed by as 
much as 0.22 GPA points from those of similar students at other high schools, but the pattern was not 
consistent; they were higher than in some years and lower in others. Thus, while there were improvements 
over time in eighth-grade graduates’ performance in high school, we do not have evidence that this was 
due to the supports from the initiative vs. other changes occurring in the district over time. 
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Table 12. NLCI Eighth-Grade Graduates’ Ninth-Grade GPAs Were Not 
Significantly Different from Comparison Schools 


Average 9th-grade GPA 


8th-Grade 9th-Grade School NLCI vs. Comparison 
for NLCI 8th-grade A : 
School Year Year Difference Estimate 
graduates 
2012-13 2013-14 1.86 -0.22 
2013-14 2014-15 2.00 0.15 
2014-15 2015-16 1.90 -0.07 
2015-16 2016-17 2.14 0.00 
2016-17 2017-18 2.20 0.11 


Note: Coefficients come from hierarchical linear models with students nested within schools, controlling for year fixed 
effects, whether the school was managed by AUSL, and student background (i.e., gender, race, ethnicity, neighborhood 
poverty, disability status), with propensity weights applied based on school characteristics in the 2011-12 school year. 
NLCI school averages were not significantly different than comparison schools in any year. 


Perceptions of School Climate in Grades 6-8 


The initiative included programs to support students’ social-emotional development, including specific 
supports for students in foster care and students whose parents were incarcerated. The continuation of 
City Year mentors also provided classroom-based support to help students succeed in school. These 
supports could have helped students to feel more supported, engaged and connected in school and with 
school peers. To discern whether these NLCI supports led to changes in school climate, we examined 
students’ responses on the annual 5Essentials Surveys taken by students in the middle and high school 
grades across the district. About 80 percent of students take the surveys in each year.“4 For information 
about the questions that are used to measure different aspects of school climate, see the Appendix. 


Students’ Perceptions of the Environment at NLCI Elementary Schools 
Improved in Some Years 


Students at NLCI schools reported feeling more engaged in their classes in later years than at the 
beginning of the study period (see Figure 9 and Table 13). They also reported having more positive 
relationships with peers and peer support for academic work in their schools. There were particularly large 
improvements in students’ reports about school climate in the 2015-16 year, although it was not 
sustained, and then again in the 2018-19 school year. In some years of the initiative, students’ perception 


“4 Most students take the surveys during a set time while they are at school. There is considerable evidence that student 
surveys provide valid assessments of students’ perceptions of school climate. For example, Chicago students’ reports of 
climate in their school explain differences in student outcomes among schools serving students with similar backgrounds 
(e.g., Allensworth & Easton, 2007), are correlated with teacher reports on similar constructs (Steinberg, Allensworth, & 
Johnson, 2011), and are predictive of students’ grades and test gains (Allensworth et al. 2014b). 
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of the environment declined relative to the year before, or remained the same. The most consistent 
improvements were in student-peer relationships and in academic engagement, which were 0.4 standard 
deviations higher in 2018-19 than in 2012-13. In both of these areas, NLCI elementary schools showed 
consistently higher reports from 2015-16 forward, relative to student reports in 2012-13 and 2013-14. 
Overall, students at NLCI schools reported considerably better perceptions of their school environment by 
the end of the study period, relative to the beginning. There were also districtwide improvements in 
students’ perceptions of school climate in some of these years, but the improvements were larger in the 
NLCI schools. 


Figure 9. Student Perceptions of School Climate at NLCI Elementary Schools 
Improved 


School climate measures, in standard deviation units 
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Note: Data come from the annual 5Essentials surveys. Measures are standardized based on all survey takers in CPS in 
the 2012-13 school year. Averages are based on students enrolled in an NLCI or CPS schools in grades 6-8 in the 
spring of each year who participated in the surveys. District response rates ranged from 79 percent to 85 percent. 
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School Climate Measures Were Higher than in Comparison NLCI schools in 
Particular Years, but Not Significantly Higher 


There were not statistically significant differences in school climate in NLCI schools than in matched 
comparison schools in any year (see Table 13), although in 2015-16 the differences with the comparison 
group approached statistical significance (p<0.10). At the same time, the differences in NLCI schools 
relative to the comparison group in 2015-16 and in 2018-19 were sizable, with a difference of 0.378 
standard deviations in students’ reports of academic engagement, and a difference of 0.476 standard 
deviations in school connectedness in 2018-19. In this case, we are concerned that there was not enough 
statistical power, and the small number of cases prevented these fairly sizable differences from reaching a 
level of statistical significance. 


Table 2. Student Perceptions of School Climate at NLCI Elementary Schools 
Were Better Than at Comparison Schools, but Did Not Reach Statistical 
Significance 


NLCI vs. NLCI vs. NLCI vs. NLCI vs. 
School NLC Comparison NLCI Comparison NLCI Comparison NLCI Comparison 
Year Average Difference Average Difference Average Difference Average Difference 
Estimate Estimate Estimate Estimate 
2012-13 1.808 -0.011 1.943 -0.025 0.216 -0.070 1.557 -0.053 
2013-14 1.682 -0.081 1.788 -0.043 0.205 0.011 1.476 -0.077 
2014-15 2.099 -0.016 2.362 0.317 0.340 -0.088 1.831 0.153 
2015-16 2.638 0.418 2.611 0.454 0.497 0.059 2.155 0.401 
2016-17 2.035 -0.238 2.152 -0.056 0.429 -0.151 1.611 -0.391 
2017-18 2.186 -0.089 2.179 0.066 0.460 -0.067 1.754 0.018 
2018-19 2.297 0.173 2.433 0.378 0.533 0.081 2.081 0.476 
Standardized Scores 
2012-13 0.114 0.242 -0.140 0.162 
2013-14 0.043 0.141 -0.150 0.126 
2014-15 0.280 0.516 -0.025 0.288 
2015-16 0.587 0.680 0.120 0.436 
2016-17 0.244 0.379 0.057 0.187 
2017-18 0.330 0.396 0.085 0.253 
2018-19 0.393 0.563 0.152 0.402 


Note: Data come from the annual 5Essentials Surveys. Measure scores in the top half of the table are reported in a 
logit scale, so scores from different measures cannot be directly compared. For the bottom half of the table, measure 
scores have been standardized using the mean and standard deviation from 2012-13. This allows comparison across 
measures and years. Coefficients come from hierarchical linear models with students nested within schools, controlling 
for year fixed effects, whether the school was managed by AUSL, and student background (i.e., gender, race, 
ethnicity,neighborhood poverty, disability status), with propensity weights applied based on school characteristics in 
the 2011-12 school year. 38 


High School Outcomes 


The NLCI provided social-emotional supports and college advising supports for ninth-graders’ transition 
to Collins Academy High School. The supports were intended to improve students’ academic performance 
in ninth grade, as well as in the post-secondary outcomes of graduates from Collins. We begin by 
examining outcomes among cohorts of first-time ninth-graders, and then show outcomes for cohorts of 
graduating seniors. 


For each outcome, we show the performance of students at Collins relative to students across the district. 
These averages are based on the population of students in the school and the district; there is no margin 
of error. Because the background characteristics of students at Collins changed considerably from year to 
year (see Chapter 2 for details), the overall trends in outcomes tell us how students at Collins performed 
over time, but they do not provide much information on whether Collins produced stronger outcomes. 
We would expect changes in average performance to occur simply due to the changes in the 
backgrounds of students in the ninth grade and graduating cohorts. Therefore, comparisons to students 
with similar backgrounds at other CPS schools is particularly important for judging the performance of 
Collins over time. For each outcome, we conducted statistical analyses of whether outcomes were 
significantly different for students at Collins than for students at other schools in CPS who had similar 
characteristics to students at Collins. Because there were few students in many of the ninth-grade and 
graduating cohorts at Collins, the tests of statistical significance should be interpreted with caution; the 
statistical power was too low to be able to find many differences to be significant. Therefore, we focus on 
the substantive size of the differences more than the significance levels. 


Outcomes among First-Time Ninth-Graders 


The ninth-grade year is a critical time for establishing students’ success in high school. Students’ ninth- 
grade attendance, Freshman Ontrack status, and ninth-grade GPA are highly predictive of their final high 
school GPA, and their likelihood of graduating high school and enrolling in college.** The City Year 
volunteers that supported students’ academic performance in their classes, as well as programs that 
supported specific students’ social-emotional development, were NLCl-provided resources that could 
have helped students perform better in their classes in their first year of high school. 


45 Easton et al. (2017); Allensworth & Easton (2007). 
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Freshman OntTrack Rates Improved over Time; Other Ninth-Grade 
Outcomes Showed Little Change 


First-time ninth-grade cohorts at Collins showed improving Freshman OnTrack rates over the years of the 
initiative (see Figure 10 and Table 14). There was a considerable increase in Freshman Ontrack rates in 
the 2013-14 school year that was sustained. In the two years prior to the initiative, Freshman OnTrack 
rates were around 60 percent, and for the first several years of the initiative they were around 70 percent. 
There was another large increase in 2017-18, when the Freshman Ontrack rate increased to 82 percent. 
District Freshman Ontrack rates improved at the same time, but the difference between Collins and the 
district average narrowed from 15 percentage points to 7 percentage points. 


The attendance and GPAs of ninth-grade cohorts were not better in post-initiative years than in pre- 
initiative years. In the 2017-18 school year, all ninth-grade outcomes were higher than in the previous 
years of the initiative (see Table 14). 


Figure 10. Collins’s Freshman OnTrack Rates Improved Considerably over Time 


Freshman Ontrack and attendance rates 
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Note: Percentages are based on all first-time ninth-grade students at Collins or other district high schools in the fall of 
each year. 
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Freshman Outcomes at Collins Were Lower Than Similar Students at 
Other Schools 


In almost all years, both before and after the initiative, Collins students had lower ninth-grade outcomes 


than similar students at other CPS schools (see Table 14). Collins had lower Freshman OnTrack and 


attendance rates than at other schools serving similar students, and students’ ninth-grade GPAs were also 


lower by as much as 0.3 GPA points. In many of these years, the differences were substantively large, but 


they were not statistically significant given the small number of students enrolled in Collins. In 2017-18, 
outcomes improved in Collins, and Freshman OnTrack rates and GPAs were similar to students with simi 
backgrounds at other CPS schools. 


Table 14. Ninth-Grade Outcomes of First-Time Ninth-Grade Cohorts at Collins 


School Difference Difference Difference 
Venr Actual from Actual from Actual from 

Expected Expected Expected 
Pre- 2011-12 61.1% -7.2% 89.1% 5.1%*** 2.03 0.08 
NLCI 2012-13 56.2% -17.9% 81.2% -4.7% 1.78 -0.28 
2013-14 70.9% -6.7% 81.1% -6.3% 1.99 -0.07 
2014-15 72.4% -4.7% 81.0% -6.7% 1.79 -0.19 
2015-16 69.1% -7.4% 81.5% -6.3% 1.79 -0.21 
2016-17 72.9% -5.7% 80.1% -7.4% 1.84 -0.30 
2017-18 82.3% +1.3% 84.9% -2.6% 2.14 -0.03 


Note: “Difference from Expected” represents Collins's school-level random effect from the hierarchical model that 
adjusts for student demographics and baseline achievement level—this provides a comparison to students with 
similar backgrounds at other CPS schools (see the Appendix for more detail). Asterisk(s) indicate the estimate is 
statistically different from zero (*<0.05; **<0.01; ***<0.001). Because of the small number of students in the sample, 
there is little statistical power for finding a significant difference. 


lar 
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College Outcomes 


The initiative included supports for Collins students as they navigated the college search and application 
process, so that they would be more likely to enroll in college, and persist in the college in which they 
enrolled. As with analyses of ninth-grade cohorts, the background characteristics of graduating seniors at 
Collins varied considerably from year-to-year, making it important to compare outcomes to similar 
students at other schools. The school averages presented here represent the population of high school 
graduates in Collins in each year, so there is no sampling error. Comparisons to similar students include 
tests of statistical significance, but as noted in Chapter 2, in some years the number of student data on 
which the college outcomes averages are based were small, so there is little statistical power to find 
differences to be statistically significant. Therefore, we highlight the substantive size of the differences, as 
well as whether they were statistically significant. 


College Enrollment Rates Recovered from Declines that Occurred 
Immediately Prior to the Initiative; College Persistence Rates Remained 
Low 


In the years immediately prior to the NLCI initiative, college enrollment rates were declining, with Collins’s 
graduates’ enrollment in any college falling from 58 percent among 2012 graduates to 39 percent among 
2013 graduates (see Figure 11 and Table 15). Although not shown here, they were even higher in 2010- 
11, suggesting a considerable decline in college outcomes over several years prior to the initiative. They 
remained low for the first two years of the initiative, with just more than 40 percent of graduates 
attending any college in the fall after graduating high school. They increased in the third and fourth years 
of the initiative, so that more than one-half of Collins's graduates enrolled in college in the 2015-16 and 
2016-17 school years. Changes in the enrollment rate in any college were largely driven by changes in 
enrollment in four-year colleges. Enrollment rates in four-year colleges declined during the years just prior 
to the initiative through the first year of the initiative, 2013-14. They increased in each subsequent year, 
with over 40 percent of graduates entering a four-year college in the 2015-16 and 2016-17 school years. 


College persistence rates among Collins graduates who went to a four-year college fell considerably at 
the same time that four-year college enrollment declined. Persistence rates were especially low among 
graduates from the classes of 2012-13 and 2013-14. They improved among students who graduated from 
Collins in 2014-15 and 2015-16, but only about one-third or less of the Collins students who entered 
college persisted for at least two years. 
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Figure 11. Collins Graduates’ College Enrollment Rates Improved, but 
Remained Below the District Average 
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Note: Percentages are based on all high school graduates from Collins or the district in each year who enrolled in 
college the fall after high school graduation. 


College Outcomes were lower for NLCI Students than Students in 
Comparison Schools in Most Years 


College enrollment rates at Collins were substantively lower than enrollment rates of similar students at 
other schools in most years, although the differences did not reach a level of statistical significance. Two 
years prior to the initiative, four-year college enrollment rates at Collins were slightly higher than the four- 
year college enrollment rates of similar students at other CPS high schools. They declined in the year 
before the initiative and were lower compared to similar students in other schools for several years, until 
improving to be slightly higher than similar students in 2015-16. During the years of the initiative, college 
enrollment rates, and enrollment rates at four-year colleges, were only higher than those of comparable 
graduates at other CPS schools among graduates in the 2015-16 school year. 


In all years, college persistence rates were lower for Collins graduates than for students from other CPS 
schools with similar backgrounds. The differences were not significant in any year because there were so 
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few students who enrolled in a four-year college in each year that there was little statistical power. For 
example, there were 45 graduating seniors in the Collins class of 2014-15, and 27 percent of them 
enrolled in a four-year college, which means the four-year college persistence rate for that year is based 
on 12 students. However, across all of the years, the college persistence rates of students from Collins 
were lower than the college persistence rates of students who had similar characteristics as Collins 
graduates but graduated from other CPS high schools. 


Table 15. Post-Secondary Outcomes of Collins Graduating Classes Were Lower 
Compared to Students with Similar Backgrounds at Other CPS Schools in Most 
Years 


Gradusun Difference Difference Difference 
Year o 9 Actual from Actual from Actual from 
Expected Expected Expected 
Bra 2011-12 58.2% +1.7% 31.6% +0.8% 40.0% -1.5% 
NLCI 2012-13 38.9% -9.4% 23.6% -3.1% 23.5% -6.6% 
2013-14 41.2% -4.6% 17.6% -5.3% <10% -11.8% 
NLCI 2014-15 42.2% -5.6% 26.7% -4.0% 33.3% -1.0% 
Years 2015-16 59.4% +1.1% 45.3% +4.2% 21.6% -8.2% 
2016-17 52.2% -6.7% 41.3% -0.5% N/A N/A 


Note: “Difference from Expected,” which is on percentage point scale, represents Collins's school-level random effect 
from the hierarchical model that adjusts for student demographics and baseline achievement level—this provides a 
comparison to students with similar backgrounds at other CPS schools (see the Appendix for more details). Asterisk(s) 
indicate the estimate is statistically different from zero (*<0.05; **<0.01; ***<0.001). The actual persistence rate for 
2013-14 is obscured because it is close to zero and has the potential to breach confidentiality if displayed. 
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Interpretive Summary 


School performance levels are strongly related to the economic context of the families they serve— 
families with more economic resources are able to invest in education in ways that produce inequitable 
outcomes. Changing these patterns so that schools serving students from neighborhoods with fewer 
economic resources have equivalent outcomes to schools serving students from more affluent 
neighborhoods is a policy challenge that has not yet been met. The NLCI aimed to improve student 
outcomes in five low-performing schools in one neighborhood by providing resources for additional 
supports across grade levels, spanning from pre-k through college choice, creating a continuum of 
supports for students as they moved from one grade to another. The hope was that this would lead to 
continual improvement in student outcomes as students would move through grades having received 
ever-increasing years of additional supports—high school supports building on elementary grade 
supports, building on pre-k supports. 


The NLCI schools did show considerable gains on all of the student outcomes studied, but only the 
substantial improvements in pre-k attendance could be attributed with confidence to the initiative. The 
schools also improved more than the district in terms of elementary attendance rates, school climate, and 
the success of eighth-grade graduates when they transitioned to high school. There were improvements 
in test scores and high school outcomes, but they remained below district averages, and did not improve 


more than other schools in the district. 


NLCI schools showed considerable improvements in student outcomes—but it would be 
difficult to outpace the gains being made districtwide. Over this period of time, the district 
implemented a number of new policies and supports, with funding often prioritized for schools with 
similar student populations and performance levels as the NLCI schools. Thus, the NLCI supplemented 
broader changes occurring in the district. Over this time period, CPS increased funding for social- 
emotional programs, supports for trauma, and restorative justice programs as an alternative to 
suspending students. The district implemented trainings in math, English, and science instruction around 
the Common Core and Next Generation Science Standards, with math instruction prioritized in schools 
like those serving NLCI students. They introduced a new teacher evaluation system that provided more 
concrete feedback on educational practice. And many schools across the city also received support from 
programs to support the high school choice process, and mentorship programs such as City Year. As a 
result, many schools in Chicago showed improving student outcomes. With the exception of pre-k 
attendance rates, we cannot attribute the gains that were made in NLCI schools to the initiative. We can 
say that districtwide changes, along with the investments from the initiative, together led to improved 


outcomes for students. 


Whether the improvements in the NLCI schools were mostly a result of district policies, the efforts of the 
school management organization, or the initiative, there were substantial improvements in many 
outcomes. The NLCI schools served students from lower-income neighborhoods than average for CPS, 
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and attendance rates tend to be correlated with income, but attendance rates in NLCI pre-k schools 
surpassed those of the CPS average in the first year of the initiative and remained higher for the following 
years. Attendance rates in grades K-2 started out lower and were the same or higher than the district 
subsequently. This matters considerably, as attendance is a stronger predictor of success in high school 
than test scores.*° Middle grade students in NLCI schools also reported much stronger school climate 
over time, surpassing district averages in terms of engagement, connectedness and peer support for 
academic work, and approaching the district average in terms of student peer relationships. Thus, 
elementary and middle grade students were increasingly engaged in school and felt more positive about 
their relationships with other students and adults in their schools. The location of the schools did not 
change, and the backgrounds of students in the NLCI elementary schools were very similar from year-to- 
year. Students were more engaged and positive about school relative to students with similar 
backgrounds who were in the schools in prior years and compared to other students in the district who 
lived in more affluent neighborhoods, on average. 


Test scores also rose considerably in the NLCI schools, but they did not increase more than the district as 
a whole. Prior studies of school turnaround have likewise shown that intensive interventions can lead to 
significant improvements in schools with prior low performance, but even with significant improvement, 
large differences often remain relative to average district schools.*” Over this time period, the annual 
gains in CPS test scores were estimated to be at about the 95th percentile among districts nationwide.*® 
The large improvements in the district overall meant NLCI schools would have to show truly exceptional 
gains (i.e., higher than gains at the 95th percentile) to be significantly higher than the other schools. 


Long-term benefits from vertically-integrated supports are attenuated by mobility and take 
time to accrue. Ultimately, only a small proportion of students experienced continuous supports across 
grade levels as a result of the initiative. Even when there were improvements in outcomes at all of the 
grade levels, the gains could only build from one year to the next to the extent that students continued 
on in NLCI schools from one year to the next. It takes many years to see a combined effect from programs 
that target different grade levels, as it takes time for students to progress from one grade to another. 
When many students transfer to other schools in the course of progressing across grades, while new 
students transfer in, the potential impact of the supports at earlier grades on outcomes in the later grades 
is decreased proportionately. 


Issues around mobility and changing populations of students in the schools are intertwined with district 
policies around school choice; there is an inherent tension between goals for supporting neighborhood 
schools in all communities and promoting opportunities for students to attend schools that are higher 
rated than their community school. NLCI schools were lower performing than many other schools that 
families might consider, in part, because the school ratings are strongly defined by average test scores, 
which are strongly related to the economic status of families. The district encourages families to choose 
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schools that are high-performing, and families want their students to go to high-performing schools that 
might offer them more educational opportunities. Yet, if families choose schools consistent with the 
district goals around school choice, there is less stability for neighborhood schools, especially those in 
neighborhoods with the fewest resources. 


These issues are particularly salient when supporting students around the high school choice process. A 
neighborhood cluster model assumes that students will move from a cluster elementary school to a 
cluster high school. But for school staff advising students about where to attend high school, there is not 
always incentive to support a stable feeder pattern, since they are also trying to encourage students to 
attend high-performing high schools. Thus, multiple initiatives with similar goals to improve educational 
opportunities for students in the most economically disinvested communities can end up conflicting with 
each other. 


NLCI turnaround strategies are intended to provide the funding to support schools in neighborhoods with 
chronically low performance to make up for inequitable opportunities that result from long-standing 
structural inequalities associated with race and income. Two million dollars in extra funding a year—which 
is approximately the level of funding from this initiative, and in the Promise Neighborhoods funded by the 
U.S. Department of Education—is a substantial investment, but it is less than 5 percent of the operating 
budget for five schools. How much extra funding is sufficient to compensate for the substantial 
differences in resources that families and communities in different parts of the city are able to invest in 
education, if opportunities are really to be equitable in all neighborhoods? Taking on the challenge of 
improving neighborhood schools that are chronically low performing to reduce inequalities with district 
averages requires planning for mobility and a potentially shifting population of students, along with 
supports to make up for the considerable inequities in family resources. 
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Appendix. Data and Analysis 
Strategies 


Data on Student Outcomes 


Data on student outcomes come from students’ administrative records, through a data sharing agreement 
with Chicago Public Schools (CPS). Data on neighborhood characteristics come from the 2015 U.S. Census. 
Each of the variables examined in this report is described below, followed by a table that provides the 
number of students that were included in the analyses. 


Pre-k Attendance among Four-Year-Old Pre-K Students 


The pre-k attendance analyses used all students in pre-k at age four. A pre-k attendance rate is equal to 
the number of days a student was marked as present, divided by the number of days that student was 
enrolled during that school year. 


Table A.1. Number of Pre-K Students in Analysis of Pre-K Attendance 


School Year NLCI Schools District 
2012-13 99 15,915 
2013-14 153 14,921 
2014-15 187 14,280 
2015-16 190 13,951 
2016-17 204 12,967 
2017-18 179 12,240 
2018-19 173 12,420 
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Prior Pre-K attendance among Entering Kindergarten Cohorts 


Kindergarten cohorts include all first-time students in kindergarten. The proportion of kindergarten 
students who attended a CPS pre-k school is simply the proportion with a record of pre-k attendance 
among all students who are first-time kindergarten students. The pre-k attendance rate is equal to the 
number of days a student was marked as present, divided by the number of days that student was 
enrolled during that school year. One analysis only examines entering kindergarteners with pre-k 
attendance records the year prior, while others included all kindergarten students, as described in the 


main text. 


Table A.2. Number of Kindergarten Students in Analyses on Prior Pre-K 
Enrollment 


School Year NLCI Schools District NLCI Schools District 
2012-13 219 32,214 123 15,409 
2013-14 211 31,543 127 14,564 
2014-15 201 30,167 133 14,409 
2015-16 214 28,754 149 13,695 
2016-17 185 27,183 128 13,333 
2017-18 188 25,934 142 12,452 
2018-19 162 25,117 127 11,823 


Attendance in K-8 


Attendance was examined separately for students in each grade from K-2, and with students in grades 3-8 
together. Attendance rate is equal to the number of days a student was marked as present, divided by the 
number of days that student was enrolled during that school year. 
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Table A.3. Number of Students in Attendance Analyses 


eee ae District cee District See District era District 
2012-13 109 30,032 79 29,894 86 29,177 503 167,650 
2013-14 151 29,678 164 30,806 108 29,321 779 165,666 
2014-15 200 28,395 189 30,125 200 30,110 996 163,551 
2015-16 213 26,947 220 28,703 204 29,271 1137 162,736 
2016-17 185 25,407 195 26,931 203 27,693 1143 159,329 
2017-18 188 24,294 168 25,268 187 26,147 1041 155,827 
2018-19 162 23,287 166 24,188 147 24,528 995 152,697 


Test Scores in Grades 2-8 


Test scores were examined separately for students in second grade, and with students in grades 3-8 
together. In some of the analyses, NWEA scores were standardized within each year by grade such that 
zero represents the district average. 


Table A.4. Number of Students in Test Score Analyses 


Number of 2nd- Number of 3rd-8th Number of 2nd-Grade Number of 3rd-8th 


Grade Students Grade Students Students Grade Students 

School NLCI ae NLCl ee er NLCI Bs 
Year Schools District Schacl District NLCI Schools _ District Schools District 
2012-13 79 24,885 462 150,018 79 24,267 457 148,050 
2013-14 95 26,352 722 155,741 95 26,170 716 155,558 
2014-15 184 27,695 937 154,066 185 27,372 936 153,680 
2015-16 191 26,557 1,058 153,756 190 26,089 1,058 153,083 
2016-17 181 25,319 1,067 150,567 182 24,795 1,071 149,893 
2017-18 168 23,741 963 147,370 166 23,280 972 146,620 
2018-19 131 22,579 922 144,764 133 22,068 928 144,260 
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Students’ Reports of School Climate 


Survey data comes from the 5Essentials Surveys administered to CPS students in grades 6-8, teachers, and 
parents, on a yearly basis. The four measures and their items from the survey used in the report are listed 
in Table A.6. The measures scores are standardized based on the district mean from the 2012-13 school 
year. Table A.7 lists the survey response rates for the district and each NLC! school by year. Students were 
included in the analyses if they participated in the survey and answered the questions in the survey bank 
for a specific survey measure. 


Table A.5. Number of Students in School Climate Analyses 


School NLCI NLCI NLCI NLCI 


Year Schools Distelet Schools District Schools Disict Schools Bemis 

2012-13 175 64,466 178 65,263 184 65,713 185 65,795 
2013-14 274 65,010 274 65,748 275 65,998 278 66,094 
2014-15 293 64,595 305 65,573 328 66,140 329 66,043 
2015-16 347 64,326 355 65,227 361 65,741 361 65,629 
2016-17 346 62,255 352 63,011 349 63,448 351 63,255 
2017-18 288 60,026 303 61,100 324 61,994 321 61,732 
2018-19 320 59,282 332 60,430 345 61,608 342 61,281 


oo 


Table A.6. Items in Survey Measures Related to Student Engagement 


Measure 


School 
Connectedness 


Academic 
Engagement 


Peer Support 
for Academic 
Work* 


Student Peer 
Relationships 


Item 


How much do you agree with the following statements about your school: 


e | feel like a real part of my school 

e People here notice when I'm good at something 

e Other students in my school take my opinions seriously 
e People at this school are friendly to me 

I'm included in lots of activities at school 


How much do you agree with the following statements about your class: 
e The topics we are studying are interesting and challenging 
e |amusually bored in class 
e =| usually look forward to this class 
e | work hard to do my best in this class 
e Sometimes | get so interested in my work | don't want to stop 
e | often count the minutes until class ends 


How many of the students in your class: 
e Think doing homework is important 
e Feel it is important to pay attention in class 
e = Feel it is important to come to school every day 
e Try hard to get good grades 


How much do you agree with the following statements about students in your 


school/class? Most students in my school/class: 
e Don't really care about each other 
e Like to put others down 
e Help each other learn 
e Just look out for themselves 
e Don't get along together very well 
e Treat each other with respect 


Note: CPS students from grades 6-12 are surveyed annually in spring. Students report their level of endorsement of 


each item (e.g., “Strongly Agree”). These item responses are then analyzed through Rasch IRT model to produce 
measure scores. Measure scores are reported in logit scale, so scores from different measures cannot be directly 


compared. *Asked only for students in grades 6-8. 
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Table A.7. Survey Response Rates by School Year 


School Year District Collins NLCI: All Elementary Chalmers Dvorak Herzl Johnson 
2011-12 79% 71% 81% 97% 77% 54% 95% 
2012-13 82% 57% 74% 83% 55% 77% 80% 
2013-14 84% 75% 76% 86% 64% 78% 75% 
2014-15 83% 73% 72% 91% 82% 57% 59% 
2015-16 85% 76% 72% 76% 95% 60% 58% 
2016-17 85% 91% 69% 53% 94% 74% 54% 


Note: “NLCI: All Elementary” refers to the four NLCI elementary schools combined (i.e., Chalmers, Dvorak, Herzl, and 
Johnson, treated as one group). 


Ninth-Grade GPAs among Eighth-Grade Graduates 


To measure the outcomes of students’ high school choice decisions, we examine ninth-grade GPAs 
among cohorts of eighth-grade graduates. Students are included in the analyses if they graduated from 
eighth grade at a CPS school and stayed enrolled in ninth grade in CPS through the end of the ninth- 
grade year. Students who attended a CPS charter high school are not included, as we do not have access 
to course transcript data for charter schools in all years. GPA is the average grade received by each 
student on a four-point scale in their ninth-grade year, weighted evenly by each course attempted. 


Table A.8. Number of Eighth-Grade Graduates in Ninth-Grade GPA Analyses 


8th-Grade School Year 9th-Grade School Year NLCI Schools District 
2012-13 2013-14 45 17,521 

2013-14 2014-15 80 17,717 
2014-15 2015-16 94 16,602 
2015-16 2016-17 90 15,925 
2016-17 2017-18 99 16,152 


Freshman OntTrack Rates, Attendance, and GPAs among Ninth-Graders 


Analyses of ninth-grade outcomes include all students who were first-time ninth-graders in a regular CPS 
school. Charter schools are not included, as we do not have access to student transcripts in all years for 
charter schools. Freshman Ontrack rate is the proportion of first-time ninth-graders who have earned five 


or more full-year course credits and failed no more than one semester of a core course by the end of 
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ninth grade.*? Attendance rate is equal to the number of days a student was marked as present, divided 
by the number of days that student was enrolled during that school year. GPA is the average grade 
received by each student on a four-point scale in their core courses (i.e., English, math, science, and social 
studies), weighted evenly by each core course attempted. 


Table A.9. Number of Ninth-Graders with Freshman OnTrack Rate, Attendance 
Rate, and GPA Analyses 


School 


Year Collins District Collins District Collins District 
2011-12 81 19,100 83 19,438 83 19,232 
2012-13 100 17,956 101 18,288 98 18,064 
2013-14 81 17,557 83 17,890 81 17,677 
2014-15 124 17,852 124 18,193 122 17,942 
2015-16 65 16,672 67 16,994 67 16,773 
2016-17 53 16,314 58 16,608 54 16,389 
2017-18 57 16,800 58 17,119 58 16,888 


College enrollment and college persistence rate among high school 
graduates 


Students who graduate from a CPS high school are included in the analyses of post-secondary outcomes. 
College enrollment rate is the proportion of high school graduates who enrolled in a post-secondary 
institution in the fall following high school graduation. College enrollment, persistence, and graduation 
rates are based on data from the National Student Clearinghouse.°° College persistence rate is the 
proportion of four-year college enrollees who enrolled immediately after high school graduation and 


have been continuously enrolled in one or more four-year institutions for four semesters." 


49 Nagaoka & Seeskin (2018). 
5° Nagaoka & Seeskin (2018). 
>’ Nagaoka & Seeskin (2018). 
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Table A.10. Number of High School Graduates with College Enrollment and 
Persistence Rate Analyses 


eae Collins District Collins District 
2011-12 74 12,447 24 3,995 
2012-13 68 12,007 16 3,708 
2013-14 60 11,830 12 3,812 
2014-15 42 11,334 1 3,852 
2015-16 64 11,284 29 4,183 
2016-17 37 10,495 15 3,999 


Modeling Elementary School Outcomes 


Our goal was to know the effects of the NLCI on student achievement. To provide the most rigorous 
assessment possible, given the rollout of the initiative, we used a propensity-score matching technique. 
We created propensity weights based on school characteristics in 2011-12, to find schools that matched 
to the NLCI schools on racial/ethnic composition, socioeconomic variables, school mobility, and average 
achievement (attendance, GPA, test scores) two years prior to the initiative. We then checked for baseline 
equivalence in the outcome variables in the years prior to NLCI funding (2012-13), and found that the 
comparison schools were not significantly different from the NLCI schools on the student outcome 
measures prior to the initiative. The propensity weight method compares NLCI schools to all other schools 
in the district, but weights the contribution of other schools to the district mean based on how similar 
they were to the NLCI schools in 2011-12 using the propensity scores. Schools that are very dissimilar 
have weights close to zero and do not contribute to the comparison, while schools that are very similar 
have weights that are close to one. 


We recognized that there would be insufficient power to discern small effects of the NLCI, but that the 
methods would produce accurate estimates of the improvements that occurred in NLCI schools. However, 
they would not be generalizable to a larger population as initiative effects unless they were large enough 
to reach a level of statistical significance (p<0.05). In the end, almost all of the tests that were insignificant 
also proved to be substantively close to zero or negative. Thus, we feel confident saying NLCI schools did 
not outperform comparison schools for those tests that did not reach a level of statistical significance. The 
one exception is with student reports of school climate, which we note in the body of the report. 


We used hierarchical linear models (HLM), with students nested within schools, to conduct statistical tests 
of whether student outcomes were higher at the NLCI schools than the comparison schools in each year, 
applying propensity weights at the school level. At the student level, we included control variables for 
student characteristics in the models (gender, race/ethnicity, neighborhood poverty level, disability status, 
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and grade level) to adjust for any changes in student population in the schools over time. Our modeling 
strategy can be summarized as follows: 
1. Identify each elementary schools' probability of being an NLCI school (i.e., propensity score), 
based on school characteristics two years prior to funding (2011-12) including racial/ethnic 


composition, socioeconomic variables, school mobility, and average achievement (attendance, 
GPA, test scores); 


2. Examine outcomes in the year prior to funding (2012-13), applying the propensity weights, to 
make sure the comparisons are equivalent; and 


3. Get effect estimates from hierarchical models with all years stacked, students within schools, 
applying propensity weights, and controlling for student demographics (gender, race/ethnicity, 
neighborhood poverty level, disability status, and grade level):>¢ 


Level 1 


Outcome = BO (2012-13) + B1 (2013-14) + B2 (2014-15) + B3 (2015-16) + B4 (2016-17) + B5(2017-18) + 
B6 (2018-19) + B7 (1st year AUSL) + B8 (later year AUSL) + =Bx (Student Demographics) + r 


Level 2 

BO = GOO + G01 (D_ever_cluster) + u0 
B1 = G10 + G11 (D_ever_cluster) + u1 
B2 = G20 + G21 (D_ever_cluster) + u2 
B3 = G30 + G31 (D_ever_cluster) + u3 
B4 = G40 + G41 (D_ever_cluster) + u4 
B5 = G50 + G51 (D_ever_cluster) + u5 
B6 = G60 + G61 (D_ever_cluster) + u6 


D_ever_cluster is a dummy variable indicating whether a school was in the NLCI. The coefficients of 
interest are GO1, G11, G21, G31, G41, G51, and G61, which each represent the difference between NLCI vs. 
comparison school among students with similar characteristics. Note that we decided not to impose 
linearity on change over time. Instead, we had a dummy variable for each year, separately estimating the 
difference from comparison schools in that given year. We did this because a linear trend did not seem to 
fit the data well for all outcomes. 


Equivalence of models with outcomes over time: The analyses were conducted at three time 
periods during the latter course of the evaluation: first, with data through 2016-17, then adding data from 
2017-18, then adding data from 2018-19. With each addition of more years of data, there is the 


°2 Since we expected different patterns between pre-k, kindergarten, first grade, and second grade, we fitted a separate 
model for each of these grade levels; grades 3-8 were fit altogether because we expected these grade levels to be similar in 
their overall patterns. 
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possibility that the relationships of the covariates with the outcomes could change over time. This means 
that the relationships discerned in earlier years could seem to change retroactively. In addition, the 
applicability of the propensity scores developed with data from the 2011-12 school becomes increasingly 
questionable as more time passes, and schools that were similar potentially become more dissimilar over 
time. Therefore, each time we added another year of data, we examined whether the size and significance 
of the relationships with the outcomes changed. 


In most cases, there was little change. However, in the case of four-year-old pre-k attendance, the 
addition of data from 2018-19 did change the relationships observed in earlier years in the earlier models. 
Beginning in the 2014-15 school year, the district increased the number of full-day pre-k options available 
to students, intentionally placing programs in neighborhoods where students were predominantly low- 
income and where there was available space in schools to expand programs. Thus, there have been 
increasing investments in full-day pre-k slots at schools in neighborhoods similar to North Lawndale, and 
preferences for enrollment slots for students with backgrounds similar to those of NLCI students over 
time. In the figure in the main text, we rely on the outcomes from the models that do not include the 
2018-19 data, as we think including the most recent year of data distorts the results in earlier years. Table 
A.11 shows the results from models that include data from different years. While the NLCI schools 
continue to show higher attendance than the comparison schools, the differences in prior years become 
smaller and non-significant when data from the most recent school year is added. 


Table A.11. Alternative Model Results for Four-Year-Old Pre-K Attendance Rates 


Schosl NLCI NLCI — NLCI ves NLCI VES 
Vour peerage eo Tuell Sen PELNe eT Sen Ea 
Difference Difference Difference 
2012-13 0.869 -0.007 -0.009 -0.021 
2013-14 0.911 0.036 0.038 0.036 
2014-15 0.905 0.053* 0.055* 0.045 
2015-16 0.918 0.039* 0.043* 0.031 
2016-17 0.920 0.041 0.045* 0.032 
2017-18 0.920 0.043* 0.030 
2018-19 0.906 0.020 


Note: Asterisks indicate the estimate of the difference relative to comparison schools is statistically different from 
zero (*<0.05; **<0.01; ***<0.001). Coefficients come from hierarchical linear models with students nested within 
schools, controlling for year fixed effects, whether the school was managed by AUSL, and student background (i.e., 
gender, race, ethnicity, neighborhood poverty, disability status), with propensity weights applied based on school 
characteristics in the 2011-12 school year. 
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Modeling NWEA Test Score Gains 


To compare NWEA test score gains between NLC! elementary schools and the comparison group, we 
used the same modeling strategy as above, except the following: 


e Test score gains are not equivalent at all points of the scale. Thus, our models control for 
students’ baseline ability (i.e., their prior-year test scores at level 1). To do this, we classified 
students’ prior-year test scores into deciles and controlled the indicator variable for each decile; 


e Since NWEA was first administered in 2013, prior-year NWEA data does not exist for 2012-13 and 
we had to exclude 2012-13 from modeling; and 


e For more efficient estimation, we chose to fit a separate model for each of grades 3-8. Test score 
gains can vary by grade level and test form (by year), and this improved the precision of the tests 
for each grade. 


Modeling High School Outcomes 


Since there was only one high school in the NLCI (Collins), we cannot apply the elementary school 
modeling strategy to examine Collins students’ performance. Hence, we employed a different modeling 
strategy. Controlling for student backgrounds (gender, race, ethnicity, neighborhood poverty level, and 
disability status), we used hierarchical linear models, with students nested within schools, to produce a 
school level residual (random effect) for each school in the district in each year. The school-level residuals 
showed the degree to which outcomes at each high school were different from the average school in the 
district, controlling for the backgrounds of students served at the school. We examined whether Collins's 
random effect was different from zero to indicate whether Collins students did significantly better or 
worse than their peers of similar backgrounds. 


Outcomes of interest included Freshman Ontrack status of first-time ninth-grade cohorts and college 
enrollment of graduating classes. For estimation purposes, we chose to fit a separate model for each 
school year of interest. Hence, the model is specified as follows: 


Level 1 

Outcome = Bo + B: (Student's 8th-Grade Test Score*?) + EB, (Student Demographics™) + r 
Level 2 

Bo = Goo + Uo 


The statistic of interest is Uo, which is the random effect of each school. We used each school’s posterior 


variance. 


°3 We used standardized ISAT/NWEA math test scores. 
>4 These include gender, race/ethnicity, neighborhood poverty level, and disability status. 
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